Dependency Hell

Hello, my name is Ian Jones, and this is the RustyElm microcast, a short, very ad-hoc journal of my attempt to learn Rust and Elm in order to build a side project.

So, I’ve finally had some time over the last week or so to get stuck into my Rust and Elm project, starting by trying to get the Rust based server up and running.

Getting a server up and running with a simple “Hello World” is a doddle with Rocket, and I had fun playing with the basics of request guards etc. to easily handle calls to /hello/ and similar play things to get a grip on how it hangs together.

However, I then made the mistake of jumping into taking a look at how to use the Juniper crate along with Rocket for creating a GraphQL server. Things very quickly came unstuck as I tried to work through a seemingly endless amount of errors that frankly made no sense to me as I have very little experience with the Rust compiler. I tried following the quick start guide but even that had problems when implemented in Rocket with slight modifications as per the integration docs, and its example on GitHub. The problems seemed to be related to dependencies, I could never get the derive macros to work properly.

Eventually I came to my senses and decided that maybe I should attempt setting up a couple of simple REST routes that return JSON first, that’s built into Rocket, and in the simple case doesn’t need hardly any dependencies.

It worked fine, changing the existing play routes to return JSON was easy. Tiny steps for the win!

Then it was time to connect up a database and grab some data from it. I thought it would be relatively simple, as again, I was going to use functionality that is effectively built into Rocket for accessing a database via the Diesel ORM.

I have some data I can use from an existing prototype of the application I’m building, the prototype currently uses an SQLite database and I have a couple of shell scripts and accompanying SQL scripts built that can extract the data and import it into a CockroachDB database. I want to use CockroachDB because it uses the PostgreSQL wire protocol and is mostly SQL compatible with PostgreSQL too, so is easy to use with most languages and frameworks, but can easily scale and have data GEO partitioned. I don’t need that scale out capability or GEO partitioning just now, but seeing as Cockroach is a breeze to set up and frankly just cool to play with, I’m going to use it. Have I mentioned that this project is all about learning how to use technologies that I find very interesting and see a great future for?

Anyway, installing the Diesel CLI was dead easy, a simple cargo install diesel_cli and we were done as I had already made sure I had the required PostgreSQL client bits and bobs installed as per the Quick Start Guide. And the basic setup of its dotenv file along with similar configuration in Rocket.toml was a no-brainer too. A quick run of diesel setup and I had the expected new migrations directory and file in my project.

Before getting stuck into using Diesel with Rocket I took a detour to make a few tweaks to my data migration scripts to use UUID columns instead of plain integer style serial columns. It’s highly recommended when data can be written to multiple nodes in a cluster, like with CockroachDB, to use random primary keys as it prevents data writes from being clumped together in its range based data setup. However, using UUIDs turned out to cause more dependency headaches further down the line that required some more quick learning, but we’ll come to that in a second.

Diesel’s migrations feature allowed me to convert the schema definition bits from my data migrations scripts into proper migration steps. The first step to set up a basic schema with extra “old ID” fields in tables, the next step being used for data import so I can re-create it easily from the SQLite database, and a follow on migration that then fixed up the data to convert the pre-existing integer based primary/foreign key pairs to use the new UUID setup. That last step also dropped the “old ID” fields on completion.

Then came the task of creating structs in the code that could be hooked up to the schema with derive(Queryable) macros etc. These macros allow the Diesel ORM to build queries and with the right use statements the fields from the structs can easily be used in code to form predicates. This turned out to be another source of Dependency Hell, largely due to the UUID and TIMESTAMP columns in the schema requiring new crates that exposed the underlying problem I had before.

There were a lot of errors related to traits not being satisfied one way or another, and infuriatingly it would often show trait signatures that exactly matched what I saw in the docs for the UUID or chrono crates that I was using. Every time I searched Stackoverflow or other sources for anything related, they always said that dependencies hadn’t been properly added to Cargo.toml or as use statements, and then showed me exactly what I already had. Eventually I pieced together enough information to realise that if the same crate is imported as a dependency by a couple of crates, and/or explicitly by your code, if they happen to import slightly different versions that seemingly have exactly the same function signatures etc, just by being even a debug level different, Rust will consider the traits completely different and incompatible.

I used the excellent cargo-tree plugin to inspect the dependency tree and saw these tiny differences in crate dependencies. At first I was at a bit of a loss as to what to do, then on a whim I decided to relax the version numbers I’d used in the dependencies sections of the Cargo.toml file. So instead of setting Rocket to be version 0.4.2, I instead used version 0.4, and so on for all the other dependencies. And boom, just like that, cargo test compiled and ran! I had previously set tight versions to try and match what I’d seen in the dependency tree, slightly coaxing versions I had control over to match the versions I saw as automatic imports. That was around the wrong way, a better strategy was to relax the versions a little so that Cargo had wiggle room and could recognise satisfied dependencies based on slightly looser version requirements. Not quite sure where my head was on that one, it’s obvious in retrospect, I blame late night programming!

With that I was able to get a couple of routes set up to return information in JSON format from the database based on UUID or other field types I played with. I even set things up so that custom JSON errors were returned when supplied query params weren’t as expected and so on.

I’ve seen it mentioned somewhere before that one of the hardest hurdles to get over when starting with languages like Rust is simply getting past the endless stream of compile time errors you encounter when first trying to find your feet. I very much felt that pain over the last few days, and I expect it’s one of the reasons Elm tries so hard to have nice helpful compiler errors that try to suggest fixes. To be fair, in a lot of cases there were helpful hints from the Rust complier too, there were times where I knocked out a big bunch of errors by following the tips it showed.

So I now had the basics of a REST-API server set up, albeit querying a single database table, but alas my GitLab CI pipeline was failing due to the new database stuff. How I got that fixed up and cut down from a run time of 26 minutes to under 3 minutes is a story for another day as I’ve run out of time.

As always, you can find me on… @ianmjones

Twitter: @ianmjones

Almost Back In The Saddle

Hello, my name is Ian Jones, and this is the RustyElm microcast, a short, very ad-hoc journal of my attempt to learn Rust and Elm in order to build a side project.

I had a draft podcast way back in mid September 2018, but never finished it as I was just too damn busy.

Things have been kinda crazy, starting with working on multiple huge releases of WP Offload Media, and helping out with WP Offload SES. Giving my best 8 hours or more of mental energy a day to my work for Delicious Brains doesn’t leave much in the tank for late night hacking. I very much enjoy working on WP Offload Media, it’s an awesome plugin, and I enjoy having primary responsibility for it’s development, so I naturally put a lot of my energy into it and am always thinking about it.

Having said that, I did get an itch for something I needed to build that also took my eye off of RustyElm and learning Rust and Elm, that being: Snippet Pixie.

Snippet Pixie is “Your little expandable text snippet helper”, for Linux. I couldn’t find a good Linux application alternative to the likes of Text Expander or Dash (which I currently use on macOS). So, in the end, I kinda just had to build it, otherwise known as scratching my itch. My primary desktop Linux distribution is elementary OS, so that’s what I built it for, in Vala.

Turns out that may have helped me get my shrivelled little brain around the Rust concept of Ownership, as Vala also doesn’t let you throw objects around without taking into consideration ownership. It caught me out a few times when the complier complained about not being able to assign an unowned reference etc, and reminded me a lot of how Rust handles ownership. Even though things aren’t exactly the same, I’m going to chalk that one up as some progress in my learning of Rust, alright?!

Anyhow, Snippet Pixie recently had a momentous release that included date/time, clipboard, embedded snippets and cursor placeholders, adding a whole new level of functionality. And even though I’m in the middle of trying to get a snap sorted so it’ll be usable on many more Linux distributions, and still have plans for more improvements to Snippet Pixie, I feel like I no longer have a huge pressure to keep adding functionality and can relax a little bit on it.

As such, I’m starting to look at getting started with my Rust and Elm based project for my evening recreational programming. Stay tuned. @ianmjones

Twitter: @ianmjones

Slow Going

  • Huge News: Elm 0.19 released!
    • Faster compiler with smaller output
    • Core packages now in “elm” namespace rather than “elm-lang”
    • New Browser module replaces HTML.program stuff
    • New Time module looks nice
    • Looking forward to using Elm again, but first need to get basic Rust backend running…
  • Progress update (reading The Rust Programming Language book):
    • Tests in Rust look great
      • Unit tests in same file as code makes sense
      • Testing private functions FTW!
      • Separate integration tests for public library functions makes sense
      • Skinny and fat files
    • Read through the I/O Project chapter
      • Impatient to get to Functional Language Features
      • But still refreshed previously learnt concepts nicely
      • Introduced more new stuff than expected, glad I read it
    • Finally got to Functional Language Features chapter
    • Just started reading the More About Cargo and chapter
      • At some point may have crate I’d like to release, who knows?
  • Released minor update to WP Cron Pixie
    • Now built with local to project Elm 0.18 tools
    • Cleaned up package dependency that prevents Elm 0.19 upgrade
    • Next version will therefore use Elm 0.19 @ianmjones

Twitter: @ianmjones

The rustyelm microcast is now available via Apple Podcasts and any other podcast app that uses the iTunes Podcast directory!…

Alright, Let’s Do This! @ianmjones

Twitter: @ianmjones

Decisions, decisions!

Hello, my name is Ian Jones, and this is my very first microcast.

I’m feverishly trying to learn Rust and brush up on Elm so that I can build a project with those two programming languages.

It occurs to me that it might be worthwhile trying to record my fumbling progress in the form of a microcast.

It would be very ad-hoc, no schedule, very much a journal for myself.

It’ll initially cover how I get on with learning Rust as I plan to build out a basic backend for the product first, then build out a basic frontend in Elm and iterate on both from there.

There will likely be rumination on the actual product idea too.

So I wonder, would anyone else be interested in listening, or should I just keep this journal to myself?

Decisions, decisions! 🤔 @ianmjones

Twitter: @ianmjones