Ryan Cromwell

Using Terraform to automate the creation and deprovisioning of AWS Member Accounts

2021-08-06T00:00:00Z

We automate provisioning developer AWS sandbox accounts for our team at Sparkbox. These accounts are paid for by the parent organization. This gives our team full access to learn and prototype with all of AWS' services. We use Billing threshold notifications to keep things from spinning out of control.

It's relatively easy to create the organizations, users, and AWS accounts. The code fits in a nice module. The challenge we face comes when a team member leaves. When they leave, we need to quickly remove access to Sparkbox resources per our security policy.

Terraform makes this seeming simple:

Remove the user from teammembers.auto.tfvars.
Push the commit to Github
Terraform Cloud kicks off a new run, planing and auto applying the changes necessary

Simple! User, policies, login profile, etc are all destroyed.

Remove Access to Sandbox Accounts

Unfortunately, consolidated billing throws a wrench in the situation. Deleting an account really just removes it from the organization. To do this, the account must have a payment method configured. This isn't an option to automate.

Our goal is to remove former team members' access to Sparkbox resources. Our Terraform automation was conflating this with deleting their accounts. It's reasonable think that, but not necessary. So instead, we'll separate the concepts of enabled users that should have access from all users that we have created accounts in our Terraform code:

locals {
  enabled_user_names = [
    for user in var.users : user.name
    if user.is_enabled
  ]
  all_user_names = toset([
    for user in var.users : user.name
  ])
}

Now we can use these individually local variables for their unique purposes:

Users that should exist:

resource "aws_iam_user" "user" {
  for_each = toset(local.enabled_user_names)
  name = each.value
}

Accounts that should exist:

resource "aws_organizations_account" "account" {
  for_each  = local.all_user_names
  name      = "Dev Sandbox ${each.value}"
  email     = "${each.value}@heysparkbox.com"
  role_name = "Administrator"
  parent_id = aws_organizations_organizational_unit.dev_team_ou.id
}

With this setup, former Sparkboxers no longer have accounts, login profiles, or any access to Sparkbox resources. We can still gain access to their sub account for whatever purpose we might have. And, if we choose, remove and delete the member account.

Making the Leap to Tech Lead

2021-06-23T00:00:00Z

Building a healthy, sustainable development organization involves defining technical leadership. In 2019, I shared my thoughts on the tech lead role at the Solidus conference in Utah. Shortly after, I hiked City Creek Canyon and fell in love with the area.

Traits of a Build and Deployment Pipeline

2021-06-23T00:00:00Z

I believe that a team's release cadence has an incredible impact on their success and enjoyment. I wrote a little about the traits that make up a successful build and deploy pipeline for our team at Sparkbox.

Architecting the Progressive Web

2017-08-26T00:00:00Z

Building a great web is something the Sparkbox takes very personally. We built what we believe is one of the first RWD e-commerce sites. We constantly battle to improve performance while delivering a great experience, even for those with a terrible connection. We know we will always be swimming up stream, but today, we believe we have a reasonable architecture for building a progressive web experience that is blazing fast and works for even the most lame devices and connections.

The crazy thing about this architecture is that it’s not terribly interesting. Essentially, we let the web do it’s thing. Javascript not happing, because you stepped into an elevator or happen to run a super old version of IE (we made that choice)? No worries. We server render all the things, accept form POSTs for all mutating interactions, and serve up PNGs instead of SVGs. Time to First Interaction a little slow? No problem – we’ll disable certain elements to save up for the important stuff.

We’ve been exploring parts of this architecture for a few years now, but this past year we’ve been able to put it all together. Throughout the next year, we’re going to be sharing how we’ve done so in the hopes that we can spur on great experiences across the web.

We forgot a Pull Request when Splitting a Repo

2016-08-31T00:00:00Z

We’re working on a little skunk works project at the office and it involved splitting a Github repository in two. After a successful split, I noticed we’d missed a PR that now impacts both repositories.

My head went to git format-patch <sha>, but I didn’t recall an option to target a file spec and at first glance it generates a patch for every commit after that SHA.

git show on the other hand definitely advertises an option to filtergit show <sha> -- path/that/moved

Format that as --pretty=email and you can direct the output to a patch for

git show --pretty=email <sha> -- path/that/moved> pr-for-other-repo.patch

Now we can pull that into our other repo with git am -3 pr-for-other-repo.patch.

Turns out that -- path/that/moved semantic does work with git format-patch. So in the end, we could have just used

git format-patch -1 -- path/that/moved
git am -3 partial-pr.patch

#startYourShift How to Make the Web Better

2016-01-29T00:00:00Z

My contribution to theShift: a writing project designed to help us produce and consume unique perspectives on topics relevant to the web industry.

A little over 2 years ago I joined Sparkbox. The focus on community, learning, and teaching were all contributors to that decision. Today we have 4 apprentices (Cat, Bennett, Brittany, and Nate) sitting among our team, double the 2014 cohort and 1 more than 2015. Build Right: Maker Series is looking to broaden our horizons by looking beyond HTML, CSS, and even tech to talk about wearables, physical health, and getting things done.

At home, we ask of our 4 awesome kids to learn an instrument, play a sport, write books or comics, and learn the skills of Scouts. They fight us all the time, but we know that breadth and awareness is as much an ingredient to success as depth.

Personally, I’ve found myself this last year “without the time” to explore things beyond my comfort zone. I’ve not shipped a personal project in over a year. This is my first personal blog post since *2014!*I’ve found a rut.

The Web is a reflection of those building it – after all The Web is a collection rather than a singular item. As a matter of building a better web, I’d like to once again make the time to expand my normal by looking outside the web.

Installing Erlang 17 for Elixir 0.13 the easy way (Mac)

2014-04-25T00:00:00Z

As of Elixir 0.13, you’ll need to have Erlang 17 installed. If you use brew to install erlang then elixir you’ll find this message awaiting you:

ERROR: OTP release R16B03-1 does not match required regex (17).*

The current Erlang brew formula is still at R16B03-1. BUT! the development version IS 17. So here you go:

brew install --devel erlang

I’m sure this will change soon, but until then this’ll get you going again.

Find Contact Shortcut in Skype for Mac

2013-11-26T00:00:00Z

It’s well understood that Skype does not advocate usability, but I won’t take it anymore. They have some shortcuts, but they’re pretty pitiful.

I’m a left hand dominant shortcut-er. Say what you will about me, but I can’t make my fingers use CMD+OPTION+F for finding a contact to start a new conversation. So I created the following shortcut.

Hello Sparkbox!

2013-10-30T00:00:00Z

Back when I started Cincy Clean Coders, it felt like the Dayton community was nearly nonexistent. It seemed like I needed to head down to the big city to attract a community of Clean Coders.

Fast forward 2 years… the Cincy group has stopped meeting, Dayton Clean Coders is going strong (with a bad-ass logo), an Elixir Group is forming, and GemCityJS is growing. There’s Build Guild and others as well. The Dayton tech scene is thriving!

It turns out the people at the center of making much of this happen are also doing great things for the web at Sparkbox by leading the world in responsive web design.

Starting next week I’m excited to join the awesome Sparkbox team, focusing hard on making the web and Dayton community an even more amazing place!

Between the platform shift to open source, Ruby, Rails, etc. and the challenge of working with some of the most passion devs I’ve met, this looks to be an exciting challenge just to keep up!

Microsoft ALM & MVP Community

This move likely means stepping away from the Microsoft ALM & MVP community. These have been some of the most passionate and hard working people I’ve met. They give an immense amount of their time towards helping build great software.

While TFS has it’s fair share of critics, they’re moving in a great direction, fast. And with a leader like Brian Harry making comments like this in private email lists…

I’m not horribly worried about being copied. If we can make the whole world provide better services, I’m happy.

…I have no doubt that set of tools will be in a great place in the coming years.

Elixir needs at least 4 JSON parsers

2013-08-27T00:00:00Z

Elixir has, at the time of this post, 3 JSON parsers hosted on expm.

Some time back I went looking for one and started exploring elixir-json, including the github repo. I loved how the encoder used protocols to implement the encoding of different types. It’s beautiful. Kudos to @carloslage.

Then I looked at the decoder. It’s full of elixir-isms… tuples, binary pattern matching, etc. Over the last few nights I decided to implement my own JSON parser learning from Carlos were I ran aground.

I don’t like the use of HashDict for objects, but spaces rule out most options like Keyword lists. I’m also not a huge fan of the nested case for dealing with the what’s next? of key/value pairs.

def parse_object( acc, << rest :: binary >> ) do
  { key, rest } = parse_content rest
  { value, rest } = lstrip(rest) |> parse_object_value 

  acc = [ { key, value } | acc ] 

  case lstrip(rest) do
    << ?}, rest :: binary >> -> { HashDict.new(acc), rest }
    << ?,, rest :: binary >> -> parse_object acc, lstrip(rest)
  end
end

Overall it feels very cohesive. You can see clearly how the pieces compose after being dispatched from the parse_content.

def parse_content( << m, rest :: binary >> ) when m in ?0..?9, do: parse_number << m, rest :: binary >>
def parse_content( << ?", rest :: binary >> ), do: rest |> parse_string
def parse_content( << ?{, rest :: binary >> ), do: lstrip(rest) |> parse_object
def parse_content( << ?[, rest :: binary >> ), do: lstrip(rest) |> parse_array

I also do nothing to support invalid JSON content as with elixir-json. Oh well, it was a learning tool.

Restoring TFS 2012 to Sandbox

2013-08-16T00:00:00Z

One of my clients recently asked for some help with getting their TFS server up to snuff. After getting the software itself in line, I needed to restore a TFS installation in a sandbox environment to plan and test out some significant changes to their Work Item data. It’s not exactly a smooth experience so lets hope this eases the day of some other poor soul. As with all things – SSDs make this a bearable duration of time.

Step 1: Install, but don’t configure TFS 2012 Update 3

TFS server components have levels of installation doneness. Since this is a sandbox, I’m not worried about scaling out and will just install a single server. To do this I:

(15 min) Install Windows Server 2012 Standard on a VM
(60 min) Windows Updates… oh mighty Zeus the Updates
(10 min) Wipe sweat from brow and walk around office cursing while fuming over what appears to be update 9,123,758 not budging
(15 min) Lay down SQL Server 2012 even though production runs on 2008 R2. Details can be found on Manually Install SQL Server for Team Foundation Server.
(10 min) Install TFS 2012 with Update 3. Don’t run the configuration wizard!!!

At this point you should have all the pieces of TFS, but nothing configured. TFS Administration console looks something like this:

Step 2: Restore TFS Databases

You can use the Administration console to restore your backups if made from a scheduled TFS backup. You could also use the tfsrestore.exe tool. In my case, I had some SQL Service permissions problems and disk space limitations that required me to manually restore each of the databases.

Step 3: Change Server IDs

Since I’ll be connecting to these in a non-isolated environment and I don’t want tools getting confused I am going to be safe and change server IDs. TFS uses Guids to identify Servers and Collections. Visual Studio and other tools using the SDKs will get confused regarding local settings and caches so this is a safeguard against black magic.

TfsConfig.exe ChangeServerID /sqlinstance:localhost /databasename:Tfs_Configuration

Step 4: Fix Accounts

My new server is not attached to any domain, let alone the one used for the production TFS. With this in mind, I need to update some accounts embedded in TFS.

TFSConfig.exe accounts /resetowner /sqlinstance:WIN-4jsv8rkkbki /databasename:Tfs_Configuration
TFSConfig.exe accounts /add /accounttype:applicationtier /account:tfsservice "/password:shhhhhh" /sqlinstance:localhost /databasename:Tfs_Configuration

Step 5: Configure TFS Application Tier

From the TFS Administration Console click Application Tier and then Configure Installed Features. This will give you a number of types of configuration scenarios. We want Application Tier only which well hook up to our now restored Tfs_Configuration database (and friends).

Make sure you choose the SQL Server to which you restored your databases (localhost for me). Once you get to the Application Tier settings page, this should show the service account we added in Step 4: Fix Accounts.

Everything should come out green upon Verification and you can let it rip. If not, invoke Google duckduckgo.com-fu.

Step 6: Update Remaining Settings

There will still be a number of settings that refer to the old topology.

Change the Urls to your new server:

In this sandbox I have no need of Reporting Services or SharePoint. I just disabled both:

Step 7: Rest

It’s certainly not straight forward, but it’s much better than in past versions.

At this point you should be able to connect to your newly restored TPC and TPs. If you find that a previously connected Visual Studio pre-selects Team Projects that were also selected in production you likely skipped Step 3. This could result in weird behavior for you in the future against production.

Using HashDict.update for Keyed Reductions (aka group by) in Elixir

2013-07-22T00:00:00Z

I wanted to start playing with Elixir’s Map and Reduce functions to get a better feel for collection transformations in the language. For this I grabbed some movie data here and planned on grabbing some perspectives.

First problem, we need to turn the data into a list of tuples. The pseudo-transformation we want to apply:

file -> lines
lines -> parts
parts -> tuples

The results ended up looking like this

[gist id=”6051117″ file=”movies.exs”]

For our data perspectives, lets start small. The number of movies per year. This is still a transformation, but it’s not going to be a one-for-one. We’re instead going to reduce the results after mapping. Why would we map? Turns out the only thing you need to know is a full list of the movie years… with dups. With that we can do an “Add or Update” to a hash for each year.

[gist id=”6051117″ file=”uniqueness.exs”]

What we’re doing is providing an entry point for count_unique which takes a collection. This creates a new HashDict which seeds our {year, count} and then recursively calls down into a variant of HashDict.update. This variant will insert a new key if not found with the 3rd parameter being the seed value. If the key is found HashDict.update will call our anonymous function to increment the value already found.

This pattern seems to work well to get the sum also. Here we map our collection to pull the year and rating. I adjusted my original to_movies to use String.to_float so that I have a numeric rating. From there I use the same HashDict pattern with the rating being our seed and accumulator.

[gist id=”6051117″ file=”sumations.exs”]

Next we’ll look at doing something a little more interesting by calculating an average and distributing the effort across nodes using ourprevious parallel map.

Triangle Kata in Elixir using Erlang processes for Parallel Maps

2013-07-17T00:00:00Z

Last time I described my setup for learning Elixir, the language built on top of Erlang with a Ruby bent. Today I wanted to get a hang of processes and mailboxes which are so important to both languages. To do this I chose the Triangle Kata as my background story. Take a look:

Update: You might wonder what that slow function is all about. I added that in there to clarify the parallelism. You would expect the slowest classifications to appears at the end of the resultant list.

My Learning Elixir (erlang) Setup

2013-07-04T00:00:00Z

The last Dayton Clean Coders meeting spiked my squirrel sense as Chris McCord and Matt Sears of our awesome host Little Lines talked about their excitement for Elixir. Of course, I’ve hung my several dozen half baked projects up for the moment as I scratch that itch.

Elixir is a Ruby inspired language that compiles down to Erlang bitcode. You can fall back to any Erlang library and many of the syntaxes, but the language is crazy easy to pick up if you’ve tried any of the other Ruby inspired languages like Coffeescript.

Here’s my setup for walking through the Elixir Getting Started:

vim with my vim-config
iex, interactive elixir will load compiled elixir modules found in the same directory. This means I can put bigger functions, modules, protocols, etc. into *.ex files and then explore and play with them using iex. You need to run
```
r(ModuleName)
```
to reload after compiles.
Guard running this Guardfile for automatically compiling the *.ex files.

vim + iex + guard = flatting the curve

Moving Blob Storage Between Azure Subscriptions

2013-04-10T00:00:00Z

We recently had some delays getting some Azure accounts coordinated for a client and chose to use a team subscription instead. After some work, the delays were resolved and we had the client managed subscription in place.

The work done using the team subscription included uploading many gigabytes of sysprepped Virtual Machine images that would stand as the template for scaling up a set of services. Once the client subscription was in place, we knew we would need to move the images over.

We were a little concerned that moving between subscriptions would take a long time. Moving between containers isn’t hard, but could you rely on the same mechanism to move between subscriptions? Turns out the answer is yes if you make the blob or container public during the move.

After you pull in your Azure Subscription into PowerShell (we used Import-AzurePublishSettingsFile) you can do this pretty easily.

Here’s an example of the Posh to do this with a public to private blob using the Azure PowerShell Cmdlets:

Release Cadence: You’re Not Agile If You’re Not Shipping

2013-03-08T00:00:00Z

I’ve spent the last 3 years exploring the in’s and out’s of different Agile frameworks and methodologies. I’ve worked with large commercial firms, government contractors, and small ISVs to adopt one set of Agile practices or another.

In the beginning things are generally amazing. There’s a new level of transparency, excitement and shared motivation. At some point, though, the question always comes up: Are we ready to ship?

This is the moment where organizations either head down the Scrum-erfall path or learn what Agile was meant to be:

…satisfy the customer
through early and continuous delivery
of valuable software.

The Path to Scrum-erfall

Unfortunately, too many organizations have taken hold of the rules and ceremonies of Kanban, Scrum, XP, and the rest, but ignore these two principles:

Working software is the primary measure of progress.
Deliver working software frequently, from a couple of weeks to a couple of months, with a
preference to the shorter timescale.

Does Shipping Heal All Wounds?

On March 4th, I opened what I hope will replace the State of Agile report: Release Cadence Report.

This survey and publication was created to refocus our industries attention back on shipping software frequently to the delight and benefit of our customers, team members, and companies. It is an attempt to relieve our software developers of ineffective anecdotal stories of Facebook and Flickr shipping dozens of times per day. Instead I want you to have the evidence to take to leaders in your organization so that you can have a serious conversation about the implications of release cadence.

Please take 10 minutes and complete this survey and you will receive an early copy of the report.

Take the Survey

You can also get yourself or your organization listed as being part of this movement by referring your coworkers, peers, friend. Sign up here and be a part of bringing shipping back to Agile.

Creating an Offline Web Platform Installer for Service Bus 1.0

2013-01-02T00:00:00Z

One of my clients is evaluating a number of Service Bus tools. One of those includes the Azure Service Bus which is now available for on-premise deployment.

Service Bus 1.0 is available only through Web Platform Installer at this time and it’s not readily apparent how you might use WPI for a network isolated environment. Hopefully this will help:

WebpiCmd.exe to find the Service Bus AppId

Web Platform Installer comes with a command line tool called WebpiCmd.exe. With it you can list and install packages. You can also create an offline WPI feed which is what we’ll do.

To list available products at the default Microsoft feed:

webpicmd /List /ListOption:Available

Somewhere in that mess is Service Bus 1.0 and it’s AppId which we’ll need to create a private, offline cache of the product and it’s dependencies. I’m using Console2 with a PowerShell tab so I can do this:

webpicmd /List /ListOption:Available | ? { $_.Contains(“Service Bus”) }

Since WebpiCmd is just writing dumb lines of text, I can’t interrogate the Product attributes, so I’m using the where-object, aliased to ?, to see which line contains the string “Service Bus”. WebPI really should be a better PS citizen.

Creating an Offline Feed

WPI works against feeds, so we’re going to create our own feed just for this product. You might maintain a growing list of products in this feed, but that’s beyond this post. To create your offline feed we’ll use the (drum roll) /Offline option:

WebpiCmd.exe /offline /Products:ServiceBusBeta /Path:C:\wpi-offline

NOTE: The Service Bus 1.0 entry in WPI seems like it has a problem with the App Fabric dependency. I’m told that’s fixed now.

Installing from the Offline Feed

Once everything is resolved and downloaded, you can use it with WPI GUI or WebpiCmd.exe.

You can not automate-all-the-things with:

webpicmd /install /Products:ServiceBus /xml:c:\wpi-offline\feeds\latest\webproductlist.xml

Or do the GUI thing by checking out WPI options:

Adaptive Problems Require Responding to Change Over Following a Plan

2012-10-22T00:00:00Z

Feedback was recently posed on the Scrum.org community forums regarding the Webster-ish definition provided for Scrum by Ken and Jeff. If you don’t recall from the Scrum Guide, the Scrum Overview definition reads:

Scrum (n): A framework within which people can address complex adaptive problems, while productively and creatively delivering products of the highest possible value.

Mario Čop, in his feedback, suggests problems cannot be adaptive. He provides some definitions of his own to remove the subjective influences around the words “adapt”, “adaptive” and “problem.” In the end, Mario says:

I’d say that the process of solving problems can be adaptive. When solving problems you can have adaptive approach and adaptive execution of it, but it makes no sense saying that problems itself are adaptive.

Whether successfully conveyed or not, the intent is that complex problems are not always clearly understood. As we approach and attempt to solve a complex problem, our understanding of that problem becomes clearer. What this means is that as we examine a problem, our understanding of it adapts. In this way we must reevaluate both the problem and the solution. Continuous planning allows for this dynamic where a plan alone constrains learning.

Scrum’s Sprint, as I’ve said before, provides a balance between space for creative problem solving and focused decision making. The outcome of a sprint allows us to reevaluate our understanding of the problem space and whether we are taking the product and organization in the appropriate direction. Here we see how we can utilize the Scrum framework to embody the Agile value of Responding to change over following a plan.

Passive Aggressive Commit Messages

2012-09-27T00:00:00Z

If anyone tries to tell you Twitter is a waste of time, they have the wrong circle of followers. Today I posted this gist somewhat facetiously asking if the following was an acceptable commit message.

Refactoring done earlier didn’t execute acceptance tests. This makes me wonder why we even have acceptance tests. The acceptance tests (and the bug they surfaced) are fixed. Ugh. Professionalism people. Please.

I say somewhat facetiously, because I actually copied it out of my almost submitted commit command. I was really close to sending it and that would have probably been the end of it. It’s unlikely anyone would have noticed it, especially the person I was directing this passive aggressive comment towards.

Follow My Own Advice

I’ve blogged recently about recognizing signals (here and here). Obviously, I’m not following my own advice. I need to go and find out why this person doesn’t care about the acceptance tests and either convince them they are important or find a better way.

Uncomfortable Part

Rather than be passive aggressive in silence, I need to have the discussion. It’s not going to be comfortable, but I have something we can both agree on: we both want this to be successful. If we can keep the conversation focused on the success of this project, we can find an understanding. If I use the same tone in our conversation that I’m using in this commit message, well, it’s not going to go well.

Fortunately, I wear big boy pants (when I go to the office) and I’m willing to get over a little uneasiness for the sake of my creation. Do it a few times and it gets much easier.

Simple Solution

It turns out this was able to exist in our codebase for a few days, because we aren’t running our acceptance tests as part of the build. I plan to remedy that tomorrow.

Scrum Fundamentals Recording Available

2012-09-27T00:00:00Z

Leading up to an open enrollment Professional Scrum Foundations on October 10th, I gave a 1 hour lunch webinar on the fundamentals of Agile and Scrum. I’ve just published this out on Youtube for anyone to watch. Let me know what you think.

tl;dr

There is a lot of misinformation out there that comes with a strong movement like Agile. It’s been coopted by some, but the vast majority see these opportunities as a way to make sure the 8 hours they spend away from their families is spent wisely. That’s where I fit in.

With that in mind, I believe very much that Scrum, like many things, can be used for both good and bad. When used well, it provides a safe zone for sustainable, creative teamwork. When used poorly, it sucks as bad if not worse than waterfall/ad-hoc.

I don’t teach Scrum such that teams go back and simply use it to manage work. Many don’t get beyond that, but in my courses, webinars, and conversations we talk about the human factors, theory, and sustainable practices that make Scrum work. This is the legacy I want Deming, Scrum, Agility, and others to be remembered by. Not velocity and commitment.