Skip to main content
November 11th, 2022

Distributing internal development tools using Homebrew

Dieter Kunze
Dieter KunzeiOS Engineer

Introduction

When working with a large software development team, being able to quickly and easily share, update and utilise internal tools is an important practice that saves the developer time on monotonous tasks and supports the development process throughout. Here at Roam Digital, we were looking for a solution to distribute our tools internally, and Homebrew quickly surfaced as a potential solution. Homebrew is a popular package manager for macOS and Linux that allows you to install and run remote packages on your machine locally, while also tracking versioning.

After some preliminary research and tests we found that using Homebrew for private GitHub repositories is something that requires more work than we initially anticipated, and comes with the requirement for custom code (something that using Homebrew for public repositories does not provide out of the box). Since the information on this topic online was not sufficient, we felt it beneficial to share our experience, so that we might save a future developer some time.

Problem

Homebrew works great when used to distribute open source software offered via public repositories. However, since the tools we wanted to distribute are private internal tools, we required the use of private repositories. When attempting to follow the same steps used to distribute public repositories, the process fails with a 404 error because the resource we are trying to download is protected.

At this stage we decided to do some research and find out if anyone else has run into this problem in the past and could potentially offer a solution. All of the articles and blog posts we found on the topic appeared to be either outdated or incomplete, though. One article from 2018 described the steps very similarly to tapping a public repository. In 2018 a change was made to Homebrew itself that added a custom implementation built atop the Ruby ‘Curl’ download strategy that added the ability to download resources from private repositories, but this was later removed. This is important because of how tapping a repository works on Homebrew.

Homebrew works by looking for a Ruby file called a “formula” in a repository and using that to find the download URL for a .tar archive of the repository that includes the resource you wish to install to your machine. The formula also includes a checksum to make sure the .tar file downloaded is the same file the developer intended to be installed.

Our solution essentially brings back this custom download strategy and allows us to tap private repositories.

Solution

Preparing our repository for Homebrew

  1. Create a folder in your repository called “Formula”.

  2. Create a Ruby file in the folder you just created called your-tool.rb.

require_relative "../custom_download_strategy"
class YourTool < Formula
  desc ""
  homepage "https://github.com/my-account/my-repo"
  url "https://github.com/my-account/my-repo/archive/refs/tags/v1.0.tar.gz", using: GitHubPrivateRepositoryDownloadStrategy
  sha256 "c919a848b1f4b53bebb0e098253cf37d2288fcbeb818201bccfa39e4b55f176d"
  license ""
  def install
    bin.install "bin/my-tool"
  end
end

Notice the naming between the class “YourTool” and the file name “your-tool.rb”, this relationship between file name and class name matters in Ruby.

3. Head to your repository on GitHub and create a new release. Since the release is just an archive of the repo, make sure you have run your tool at least once and that the repository includes the tool executable somewhere.

4. Copy the link for the file “Source Code (tar.gz)” and paste it after “url” in the formula file.

5. Download the “Source Code (tar.gz)” file and then in Terminal, Run “shasum -a 256 <your-tar-file-location>”, then paste the checksum into your formula after “sha256”.

6. Update the tool name after “bin.install”.

7. Now for the secret sauce, the custom download strategy. Create a Ruby file in the main folder of your project called custom_download_strategy.rb.

# This is based on the following, with minor fixes.
# https://github.com/Homebrew/brew/blob/193af1442f6b9a19fa71325160d0ee2889a1b6c9/Library/Homebrew/compat/download_strategy.rb#L48-L157
# GitHubPrivateRepositoryDownloadStrategy downloads contents from GitHub
# Private Repository. To use it, add
# `:using => GitHubPrivateRepositoryDownloadStrategy` to the URL section of
# your formula. This download strategy uses GitHub access tokens (in the
# environment variables `HOMEBREW_GITHUB_API_TOKEN`) to sign the request.  This
# strategy is suitable for corporate use just like S3DownloadStrategy, because
# it lets you use a private GitHub repository for internal distribution.  It
# works with public one, but in that case simply use CurlDownloadStrategy.
class GitHubPrivateRepositoryDownloadStrategy < CurlDownloadStrategy
  require "utils/formatter"
  require "utils/github"
def initialize(url, name, version, **meta)
    super
    parse_url_pattern
    set_github_token
  end
  def parse_url_pattern
    unless match = url.match(%r{https://github.com/([^/]+)/([^/]+)/(\S+)})
      raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Repository."
    end
    _, @owner, @repo, @filepath = *match
  end
  def download_url
    "https://#{@github_token}@github.com/#{@owner}/#{@repo}/#{@filepath}"
  end
  private
  def _fetch(url:, resolved_url:, timeout:)
    curl_download download_url, to: temporary_path
  end
  def set_github_token
    @github_token = ENV["HOMEBREW_GITHUB_API_TOKEN"]
    unless @github_token
      raise CurlDownloadStrategyError, "Environmental variable HOMEBREW_GITHUB_API_TOKEN is required."
    end
    validate_github_repository_access!
  end
  def validate_github_repository_access!
    # Test access to the repository
    GitHub.repository(@owner, @repo)
  rescue GitHub::API::HTTPNotFoundError
    # We switched to GitHub::API::HTTPNotFoundError,
    # because we can now handle bad credentials messages
    message = <<~EOS
      HOMEBREW_GITHUB_API_TOKEN can not access the repository: #{@owner}/#{@repo}
      This token may not have permission to access the repository or the url of formula may be incorrect.
    EOS
    raise CurlDownloadStrategyError, message
  end
end
# GitHubPrivateRepositoryReleaseDownloadStrategy downloads tarballs from GitHub
# Release assets. To use it, add
# `:using => GitHubPrivateRepositoryReleaseDownloadStrategy` to the URL section of
# your formula. This download strategy uses GitHub access tokens (in the
# environment variables HOMEBREW_GITHUB_API_TOKEN) to sign the request.
class GitHubPrivateRepositoryReleaseDownloadStrategy < GitHubPrivateRepositoryDownloadStrategy
  def initialize(url, name, version, **meta)
    super
  end
  def parse_url_pattern
    url_pattern = %r{https://github.com/([^/]+)/([^/]+)/releases/download/([^/]+)/(\S+)}
    unless @url =~ url_pattern
      raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Release."
    end
    _, @owner, @repo, @tag, @filename = *@url.match(url_pattern)
  end
  def download_url
    "https://#{@github_token}@api.github.com/repos/#{@owner}/#{@repo}/releases/assets/#{asset_id}"
  end
  private
  def _fetch(url:, resolved_url:, timeout:)
    # HTTP request header `Accept: application/octet-stream` is required.
    # Without this, the GitHub API will respond with metadata, not binary.
    curl_download download_url, "--header", "Accept: application/octet-stream", to: temporary_path
  end
  def asset_id
    @asset_id ||= resolve_asset_id
  end
  def resolve_asset_id
    release_metadata = fetch_release_metadata
    assets = release_metadata["assets"].select { |a| a["name"] == @filename }
    raise CurlDownloadStrategyError, "Asset file not found." if assets.empty?
   assets.first["id"]
  end
  def fetch_release_metadata
    #release_url = "https://api.github.com/repos/#{@owner}/#{@repo}/releases/tags/#{@tag}"
    #GitHub::API.open_rest(release_url)
    GitHub.get_release(@owner, @repo, @tag)
  end
end

Although a few resources contributed to our troubleshooting, this article provided a solution quite similar to what we ultimately settled on. The custom download strategy code includes 2 classes, “GitHubPrivateRepositoryDownloadStrategy” and “GitHubPrivateRepositoryReleaseDownloadStrategy”. The former is used when targeting an archive generated by GitHub, and the latter when targeting a manual archive uploaded to a GitHub release.

8. Our final folder structure is as follows:

9. Install brew if you haven’t already.

10. To test your changes, open Terminal, then:

a. Run “export HOMEBREW_GITHUB_API_TOKEN=<homebrew_api_token_goes_here>”.

b. Run “brew tap <my-formula-file.rb location>”.

c. Run “brew install <my-tool>”.

d. If all went well, celebrate! 🍺

11. Once the local test passes, push your changes to your repository.

Tapping your repository

On GitHub, head to Settings -> Developer settings -> Personal access tokens generate a new token with repository permission.

In Terminal

  1. Run “export HOMEBREW_GITHUB_API_TOKEN=<homebrew_api_token_goes_here>”.

  2. Run “brew tap my-account/my-repository https://github.com/my-account/my-repo”.

Tip: If your repository name is prefixed with “homebrew”, then you can omit the GitHub URL from this step and just call “brew tap my-account/homebrew-my-repository”.

  1. Enter your GitHub username and your <HOMEBREW_GITHUB_API_TOKEN> as the password.

  2. Run “brew install <my-tool>”.

  3. If all went well, celebrate! 🍺

Hangups

Another hangup that caught us out was the confusion around wether we should tap the “Source Code (tar.gz)” file automatically created by the GitHub release, or create our own .tar archive of the tool we wanted to ship. We ultimately decided on using the archive generated by the release for easier deployment in future, but that required the use of a different download strategy than we had initially gotten to work because of the odd URL format. Both downloaded strategies are included in the code above.

Tips

  • If you have made changes to your repository and pushed them to GitHub, you will need to run “brew untap” and then “brew tap” to refresh the local repository copy before installing again.

  • Be aware that after running “brew install” you may encounter a message that the file is already downloaded. If this is undesirable because you updated your release then run “rm <file-location>” and then run “brew install” again to overwrite it.

Summary

Using a custom download strategy, we can successfully tap private GitHub repositories and offer our internal tools to our development team via Homebrew.

A few extra steps were required compared to tapping a public repository, but works great thanks to the community knowledge and team problem solving internally at Roam Digital. We got 99% of the way there given the resources we found online, but the fix that ended up working for uswas moving the custom_download_strategy.rb file to a more logical location.

We believe that Homebrew is a great solution to distributing internal tools here at Roam Digital and we are happy to share our experience in hopes that it saves you some precious development time.

Happy coding!

Dieter Kunze
Dieter KunzeDieter is an experienced developer and the founder of Mammoth Apps, where he spent 5 years creating and launching apps before joining Roam Digital's Mobile chapter.