Glorified Markov Chainhttps://www.burtonini.com/blog/2023-03-06T15:24:00+00:00Building a big-endian Arm system in Yocto2023-03-06T15:24:00+00:002023-03-06T15:24:00+00:00Ross Burtontag:www.burtonini.com,2023-03-06:/blog/2023/03/06/big-endian/<p>For reasons I won't bore anyone with I needed to build a 32-bit big-endian system with the Yocto Project to test a package, and I thought I'd write the steps down in case I ever need to do it again (or, even more unlikely, someone else needs to do it …</p><p>For reasons I won't bore anyone with I needed to build a 32-bit big-endian system with the Yocto Project to test a package, and I thought I'd write the steps down in case I ever need to do it again (or, even more unlikely, someone else needs to do it).</p>
<p>For unsurprising reasons I thought I'd do a big-endian Arm build. So we start by picking the <code>qemuarm</code> machine, which is a Armv7-A processor (Cortex-A15, specifically) in little-endian mode by default.</p>
<div class="highlight"><pre><span></span><code>MACHINE = "qemuarm"
</code></pre></div>
<p><code>qemumarm.conf</code> requires <code>tune-cortexa15.inc</code> which then requires <code>arch-armv7ve.inc</code>, and this file defines the base tunes. The default tune is <code>armv7ve</code>, we can make it big-endian by simply adding a <code>b</code>:</p>
<div class="highlight"><pre><span></span><code><span class="n">DEFAULTTUNE</span><span class="o">:</span><span class="n">qemuarm</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"armv7veb"</span>
</code></pre></div>
<p>And now we just build an image:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nv">MACHINE</span><span class="o">=</span>qemuarm<span class="w"> </span>bitbake<span class="w"> </span>core-image-minimal
...
Summary:<span class="w"> </span><span class="m">4</span><span class="w"> </span>tasks<span class="w"> </span>failed:
<span class="w"> </span>.../poky/meta/recipes-kernel/linux/linux-yocto_6.1.bb:do_package_qa
<span class="w"> </span>.../poky/meta/recipes-graphics/xorg-proto/xorgproto_2022.2.bb:do_configure
<span class="w"> </span>.../poky/meta/recipes-core/glib-2.0/glib-2.0_2.74.5.bb:do_configure
<span class="w"> </span>.../poky/meta/recipes-graphics/wayland/wayland_1.21.0.bb:do_configure
</code></pre></div>
<p>Or not.</p>
<p>There are two failure cases here. First, the kernel:</p>
<div class="highlight"><pre><span></span><code><span class="nl">ERROR</span><span class="p">:</span><span class="w"> </span><span class="n">linux</span><span class="o">-</span><span class="n">yocto</span><span class="o">-</span><span class="mf">6.1.9</span><span class="o">+</span><span class="n">gitAUTOINC</span><span class="o">+</span><span class="n">d7393c5752_ccd3b20fb5</span><span class="o">-</span><span class="n">r0</span><span class="w"> </span><span class="nl">do_package_qa</span><span class="p">:</span><span class="w"> </span><span class="n">QA</span><span class="w"> </span><span class="nl">Issue</span><span class="p">:</span><span class="w"> </span><span class="n">Endiannes</span><span class="w"> </span><span class="n">did</span><span class="w"> </span><span class="ow">not</span><span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">expected</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">modules</span><span class="o">/</span><span class="mf">6.1.9</span><span class="o">-</span><span class="n">yocto</span><span class="o">-</span><span class="n">standard</span><span class="o">/</span><span class="n">kernel</span><span class="o">/</span><span class="n">net</span><span class="o">/</span><span class="n">ipv4</span><span class="o">/</span><span class="n">ah4</span><span class="p">.</span><span class="n">ko</span><span class="w"> </span><span class="o">[</span><span class="n">arch</span><span class="o">]</span>
</code></pre></div>
<p>It turns out the kernel needs to be configured specifically to be big or little endian, and the default configuration is, predictably, little endian. There is <a href="https://bugzilla.yoctoproject.org/show_bug.cgi?id=9480">a bug</a> open to make this automatic, but big-endian really is dead because it has been open since 2016. The solution is a quick kernel configuration fragment added to the kernel's <code>SRC_URI</code>:</p>
<div class="highlight"><pre><span></span><code><span class="n">CONFIG_CPU_BIG_ENDIAN</span><span class="o">=</span><span class="n">y</span>
<span class="n">CONFIG_CPU_LITTLE_ENDIAN</span><span class="o">=</span><span class="n">n</span>
</code></pre></div>
<p>With this, the kernel builds as expected. The second set of failures are all from Meson, failing to execute a target binary:</p>
<div class="highlight"><pre><span></span><code><span class="cp">../xorgproto-2022.2/meson.build:22:0: ERROR: Executables created by c compiler armeb-poky-linux-gnueabi-gcc [...] are not runnable.</span>
</code></pre></div>
<p>Meson is trying to run the target binaries in a <code>qemu-user</code> that we set up, but the problem here is to save build time we only build the qemu targets that are typically used. This doesn't include usermode big-endian 32-bit Arm, so this target needs enabling:</p>
<div class="highlight"><pre><span></span><code><span class="n">QEMU_TARGETS</span><span class="o">:</span><span class="n">append</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">" armeb"</span>
</code></pre></div>
<p>Now the image builds successfully, and we discover that indeed gdbm refuses to open a database which was generated on a system with a different endian.</p>PySnooper and BitBake2023-01-06T11:02:00+00:002023-01-06T11:02:00+00:00Ross Burtontag:www.burtonini.com,2023-01-06:/blog/2023/01/06/pysnooper/<p>Yesterday I discovered <a href="https://github.com/cool-RR/PySnooper">PySnooper</a>, which describes itself as "a poor man's debugger":</p>
<blockquote>
<p>Your story: You're trying to figure out why your Python code isn't doing what
you think it should be doing. You'd love to use a full-fledged debugger with
breakpoints and watches, but you can't be bothered to set …</p></blockquote><p>Yesterday I discovered <a href="https://github.com/cool-RR/PySnooper">PySnooper</a>, which describes itself as "a poor man's debugger":</p>
<blockquote>
<p>Your story: You're trying to figure out why your Python code isn't doing what
you think it should be doing. You'd love to use a full-fledged debugger with
breakpoints and watches, but you can't be bothered to set one up right now.</p>
</blockquote>
<p>I know that guy! Especially when I'm debugging some Python code in a BitBake class or recipe and attaching a debugger is even more annoying than usual. I've previously written a <a href="https://github.com/rossburton/meta-ross/blob/master/classes/pdb.bbclass">tiny class</a> to start a rpdb session as needed, but I don't get on with pdb for some reason.</p>
<p>The example makes it look pretty awesome for quick debugging:</p>
<div class="highlight"><pre><span></span><code><span class="n">Source</span><span class="w"> </span><span class="n">path</span><span class="p">:</span><span class="o">...</span><span class="w"> </span><span class="n">example</span><span class="o">.</span><span class="n">py</span>
<span class="n">Starting</span><span class="w"> </span><span class="k">var</span><span class="p">:</span><span class="o">..</span><span class="w"> </span><span class="n">number</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">6</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.482187</span><span class="w"> </span><span class="n">call</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">def</span><span class="w"> </span><span class="n">number_to_bits</span><span class="p">(</span><span class="n">number</span><span class="p">):</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.482561</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">number</span><span class="p">:</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.482655</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="n">bits</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[]</span>
<span class="n">New</span><span class="w"> </span><span class="k">var</span><span class="p">:</span><span class="o">.......</span><span class="w"> </span><span class="n">bits</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[]</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.482732</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="n">number</span><span class="p">:</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.482830</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="n">number</span><span class="p">,</span><span class="w"> </span><span class="n">remainder</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">divmod</span><span class="p">(</span><span class="n">number</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span>
<span class="n">Modified</span><span class="w"> </span><span class="k">var</span><span class="p">:</span><span class="o">..</span><span class="w"> </span><span class="n">number</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">3</span>
<span class="n">New</span><span class="w"> </span><span class="k">var</span><span class="p">:</span><span class="o">.......</span><span class="w"> </span><span class="n">remainder</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.482907</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">bits</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">remainder</span><span class="p">)</span>
<span class="n">Modified</span><span class="w"> </span><span class="k">var</span><span class="p">:</span><span class="o">..</span><span class="w"> </span><span class="n">bits</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.483028</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="n">number</span><span class="p">:</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.483130</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="n">number</span><span class="p">,</span><span class="w"> </span><span class="n">remainder</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">divmod</span><span class="p">(</span><span class="n">number</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span>
<span class="n">Modified</span><span class="w"> </span><span class="k">var</span><span class="p">:</span><span class="o">..</span><span class="w"> </span><span class="n">number</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span>
<span class="n">Modified</span><span class="w"> </span><span class="k">var</span><span class="p">:</span><span class="o">..</span><span class="w"> </span><span class="n">remainder</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.483208</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">bits</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">remainder</span><span class="p">)</span>
<span class="n">Modified</span><span class="w"> </span><span class="k">var</span><span class="p">:</span><span class="o">..</span><span class="w"> </span><span class="n">bits</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.483323</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="n">number</span><span class="p">:</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.483419</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="n">number</span><span class="p">,</span><span class="w"> </span><span class="n">remainder</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">divmod</span><span class="p">(</span><span class="n">number</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span>
<span class="n">Modified</span><span class="w"> </span><span class="k">var</span><span class="p">:</span><span class="o">..</span><span class="w"> </span><span class="n">number</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.483497</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="n">bits</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">remainder</span><span class="p">)</span>
<span class="n">Modified</span><span class="w"> </span><span class="k">var</span><span class="p">:</span><span class="o">..</span><span class="w"> </span><span class="n">bits</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.483593</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="n">number</span><span class="p">:</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.483697</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">bits</span>
<span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mf">07.483773</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">bits</span>
<span class="n">Return</span><span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="o">..</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span>
<span class="n">Elapsed</span><span class="w"> </span><span class="n">time</span><span class="p">:</span><span class="w"> </span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mf">00.001749</span>
</code></pre></div>
<p>So here's my thirty second explainer on how to use PySnooper with BitBake. First, we need to install it:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pip3<span class="w"> </span>install<span class="w"> </span>pysnooper
</code></pre></div>
<p>Then you can just <code>import pysnooper</code> and decorate functions to get them annotated at runtime:</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">pysnooper</span>
<span class="nd">@pysnooper</span><span class="o">.</span><span class="n">snoop</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">some_function</span><span class="p">():</span>
<span class="o">...</span>
</code></pre></div>
<p>That's the theory, but anyone who has tried throwing <code>print("here")</code> messages into classes or recipes knows this doesn't work. They execute in a child process which doesn't have standard output connected to the console, but luckily the <code>snoop</code> function can instead write the messages to a filename or stream <em>or callable</em>, which lets us glue PySnooper to BitBake's logging:</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">pysnooper</span>
<span class="nd">@pysnooper</span><span class="o">.</span><span class="n">snoop</span><span class="p">(</span><span class="n">bb</span><span class="o">.</span><span class="n">plain</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">some_function</span><span class="p">():</span>
<span class="o">...</span>
</code></pre></div>
<p>As a working example, I added the annotation to <code>get_source_date_epoch()</code> in <code>meta/lib/oe/reproducible.py</code>:</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">pysnooper</span>
<span class="nd">@pysnooper</span><span class="o">.</span><span class="n">snoop</span><span class="p">(</span><span class="n">bb</span><span class="o">.</span><span class="n">plain</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">get_source_date_epoch</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">sourcedir</span><span class="p">):</span>
<span class="k">return</span> <span class="p">(</span>
<span class="n">get_source_date_epoch_from_git</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">sourcedir</span><span class="p">)</span> <span class="ow">or</span>
<span class="n">get_source_date_epoch_from_youngest_file</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">sourcedir</span><span class="p">)</span> <span class="ow">or</span>
<span class="n">fixed_source_date_epoch</span><span class="p">(</span><span class="n">d</span><span class="p">)</span>
<span class="p">)</span>
</code></pre></div>
<p>And now when we start BitBake, we get to see the output:</p>
<div class="highlight"><pre><span></span><code><span class="n">Source</span><span class="w"> </span><span class="n">path</span><span class="p">:</span><span class="o">...</span><span class="w"> </span><span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">ross</span><span class="o">/</span><span class="n">Yocto</span><span class="o">/</span><span class="n">poky</span><span class="o">/</span><span class="n">meta</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">oe</span><span class="o">/</span><span class="n">reproducible</span><span class="o">.</span><span class="n">py</span>
<span class="n">Starting</span><span class="w"> </span><span class="k">var</span><span class="p">:</span><span class="o">..</span><span class="w"> </span><span class="n">d</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o"><</span><span class="n">bb</span><span class="o">.</span><span class="n">data_smart</span><span class="o">.</span><span class="n">DataSmart</span><span class="w"> </span><span class="n">object</span><span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="mh">0xffff9e30dcf0</span><span class="o">></span>
<span class="n">Starting</span><span class="w"> </span><span class="k">var</span><span class="p">:</span><span class="o">..</span><span class="w"> </span><span class="n">sourcedir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'/yocto/ross/build/tmp/work-shared/llvm-project-source-15.0.6-r0/git'</span>
<span class="mi">10</span><span class="p">:</span><span class="mi">56</span><span class="p">:</span><span class="mf">57.198016</span><span class="w"> </span><span class="n">call</span><span class="w"> </span><span class="mi">156</span><span class="w"> </span><span class="n">def</span><span class="w"> </span><span class="n">get_source_date_epoch</span><span class="p">(</span><span class="n">d</span><span class="p">,</span><span class="w"> </span><span class="n">sourcedir</span><span class="p">):</span>
<span class="mi">10</span><span class="p">:</span><span class="mi">56</span><span class="p">:</span><span class="mf">57.199750</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">158</span><span class="w"> </span><span class="n">get_source_date_epoch_from_git</span><span class="p">(</span><span class="n">d</span><span class="p">,</span><span class="w"> </span><span class="n">sourcedir</span><span class="p">)</span><span class="w"> </span><span class="ow">or</span>
<span class="mi">10</span><span class="p">:</span><span class="mi">56</span><span class="p">:</span><span class="mf">57.341387</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="mi">157</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="p">(</span>
<span class="mi">10</span><span class="p">:</span><span class="mi">56</span><span class="p">:</span><span class="mf">57.341978</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">157</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="p">(</span>
<span class="n">Return</span><span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="o">..</span><span class="w"> </span><span class="mi">1669716358</span>
<span class="n">Elapsed</span><span class="w"> </span><span class="n">time</span><span class="p">:</span><span class="w"> </span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mf">00.144763</span>
</code></pre></div>
<p>Useful!</p>
<p>The default log depth is 1 so you don't see inside functions, but that can be changed when decorating You can also wrap smaller code blocks using <code>with</code> blocks.</p>
<p>The biggest catch is remembering that BitBake classes and recipes are not Python, they just have Python blocks in, so you can't decorate a function inside a class or recipe. In this case you'll need to use <code>with</code> block.</p>
<p>This looks like a very useful tool and I look forward to using it next time I'm tearing my increasingly greying hair out.</p>
<p><small>NP: <cite><a href="https://serein.co.uk/releases/brambles-charcoal/">Charcoal</a></cite>, Brambles</small></p>Faster image transfer across the network with zsync2021-06-10T16:28:00+01:002021-06-10T16:28:00+01:00Ross Burtontag:www.burtonini.com,2021-06-10:/blog/2021/06/10/yocto-zsync/<p>Those of us involved in building operating system images using tools such as OpenEmbedded/Yocto Project or Buildroot don't always have a power build machine under our desk or in the same building on gigabit. Our build machine may be in the cloud, or in another office over a VPN …</p><p>Those of us involved in building operating system images using tools such as OpenEmbedded/Yocto Project or Buildroot don't always have a power build machine under our desk or in the same building on gigabit. Our build machine may be in the cloud, or in another office over a VPN running over a slow residential ADSL connection. In these scenarios, repeatedly downloading gigabyte-sized images for local testing can get very tedious.</p>
<p>There are some interesting solutions if you use Yocto: you could expose the shared state over the network and recreate the image, which if the configurations are the same will result in no local compilation. However this isn't feasible if your local machine isn't running Linux or you just want to download the image without any other complications. This is where <a href="http://zsync.moria.org.uk">zsync</a> is useful.</p>
<p><a href="http://zsync.moria.org.uk">zsync</a> is a tool similar to <a href="https://rsync.samba.org">rsync</a> but optimised for transfering single large files across the network. The server generates metadata containing the chunk information, and then shares both the image and the metadata over HTTP. The client can then use any existing local file as a <em>seed file</em> to speed up downloading the remote file.</p>
<p>On the server, run <code>zsyncmake</code> on the file to be transferred to generate the <code>.zsync</code> metadata. You can also pass <code>-z</code> if the file isn't already compressed to tell it to compress the file first.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>-lh<span class="w"> </span>core-image-minimal-*.wic*
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>ross<span class="w"> </span>ross<span class="w"> </span>421M<span class="w"> </span>Jun<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">13</span>:44<span class="w"> </span>core-image-minimal-fvp-base-20210610124230.rootfs.wic
$<span class="w"> </span>zsyncmake<span class="w"> </span>-z<span class="w"> </span>core-image-minimal-*.wic
$<span class="w"> </span>ls<span class="w"> </span>-lh<span class="w"> </span>core-image-minimal-*.wic*
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>ross<span class="w"> </span>ross<span class="w"> </span><span class="m">4</span>.7K<span class="w"> </span>Jun<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">13</span>:44<span class="w"> </span>core-image-minimal-fvp-base-20210610124230.rootfs.manifest
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>ross<span class="w"> </span>ross<span class="w"> </span>421M<span class="w"> </span>Jun<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">13</span>:44<span class="w"> </span>core-image-minimal-fvp-base-20210610124230.rootfs.wic
-rw-r--r--<span class="w"> </span><span class="m">1</span><span class="w"> </span>ross<span class="w"> </span>ross<span class="w"> </span>53M<span class="w"> </span>Jun<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">13</span>:45<span class="w"> </span>core-image-minimal-fvp-base-20210610124230.rootfs.wic.gz
</code></pre></div>
<p>Here we have ~420MB of disk image, which compressed down to a slight 53MB, and just ~5KB of metadata. This image compressed very well as the raw image is largely empty space, but for the purposes of this example we can ignore that.</p>
<p>The zsync client downloads over HTTP and has some <a href="http://zsync.moria.org.uk/server">non-trivial requirements</a> so you can't just use any HTTP server, specifically my go-to dumb server (Python's integrated <code>http.server</code>) isn't sufficient. If you want a hassle-free server then the Node.js package <code>http-server</code> works nicely, or any other proper server will work. However you choose to do it, share both the <code>.zsync</code> and <code>.wic.gz</code> files.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>npm<span class="w"> </span>install<span class="w"> </span>-g<span class="w"> </span>http-server
$<span class="w"> </span>http-server<span class="w"> </span>-p<span class="w"> </span><span class="m">8080</span><span class="w"> </span>/path/to/images
</code></pre></div>
<p>Now you can use the zsync client to download the images. Sadly zsync isn't actually magical, so the first download will still need to download the full file:</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="w"> </span><span class="n">zsync</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">buildmachine</span><span class="p">:</span><span class="mi">8080</span><span class="o">/</span><span class="n">core</span><span class="o">-</span><span class="n">image</span><span class="o">-</span><span class="n">minimal</span><span class="o">-</span><span class="n">fvp</span><span class="o">-</span><span class="n">base</span><span class="o">-</span><span class="mf">20210610124230.</span><span class="n">rootfs</span><span class="o">.</span><span class="n">wic</span><span class="o">.</span><span class="n">zsync</span>
<span class="n">No</span><span class="w"> </span><span class="n">relevent</span><span class="w"> </span><span class="n">local</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="n">found</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">I</span><span class="w"> </span><span class="n">will</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">downloading</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">whole</span><span class="w"> </span><span class="n">file</span><span class="o">.</span>
<span class="n">downloading</span><span class="w"> </span><span class="n">from</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">buildmachine</span><span class="p">:</span><span class="mi">8080</span><span class="o">/</span><span class="n">core</span><span class="o">-</span><span class="n">image</span><span class="o">-</span><span class="n">minimal</span><span class="o">-</span><span class="n">fvp</span><span class="o">-</span><span class="n">base</span><span class="o">-</span><span class="mf">20210610124230.</span><span class="n">rootfs</span><span class="o">.</span><span class="n">wic</span><span class="o">.</span><span class="n">gz</span><span class="p">:</span>
<span class="c1">#################### 100.0% 7359.7 kBps DONE</span>
<span class="n">verifying</span><span class="w"> </span><span class="n">download</span><span class="o">...</span><span class="n">checksum</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="n">OK</span>
<span class="n">used</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">local</span><span class="p">,</span><span class="w"> </span><span class="n">fetched</span><span class="w"> </span><span class="mi">55208393</span>
</code></pre></div>
<p>However, subsequent downloads will be a lot faster as only the differences will be fetched. Say I decide that <code>core-image-minimal</code> is too, well, minimal, and build <code>core-image-sato</code> which is a full X.org stack instead of just busybox. After building the the image and metadata we now have a ~700MB image:</p>
<div class="highlight"><pre><span></span><code>-rw-r--r-- 1 ross ross 729M Jun 10 14:17 core-image-sato-fvp-base-20210610125939.rootfs.wic
-rw-r--r-- 1 ross ross 118M Jun 10 14:18 core-image-sato-fvp-base-20210610125939.rootfs.wic.gz
-rw-r--r-- 1 ross ross 2.2M Jun 10 14:19 core-image-sato-fvp-base-20210610125939.rootfs.wic.zsync```
</code></pre></div>
<p>Normally we'd have to download the full 730MB, but with zsync we can just fetch the differences. By telling the client to use the existing <code>core-image-minimal</code> as a seed file, we can fetch the new <code>core-image-sato</code>:</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="w"> </span><span class="n">zsync</span><span class="w"> </span><span class="o">-</span><span class="n">i</span><span class="w"> </span><span class="n">core</span><span class="o">-</span><span class="n">image</span><span class="o">-</span><span class="n">minimal</span><span class="o">-</span><span class="n">fvp</span><span class="o">-</span><span class="n">base</span><span class="o">-</span><span class="mf">20210610124230.</span><span class="n">rootfs</span><span class="o">.</span><span class="n">wic</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">buildmachine</span><span class="p">:</span><span class="mi">8080</span><span class="o">/</span><span class="n">core</span><span class="o">-</span><span class="n">image</span><span class="o">-</span><span class="n">sato</span><span class="o">-</span><span class="n">fvp</span><span class="o">-</span><span class="n">base</span><span class="o">-</span><span class="mf">20210610125939.</span><span class="n">rootfs</span><span class="o">.</span><span class="n">wic</span><span class="o">.</span><span class="n">zsync</span>
<span class="n">reading</span><span class="w"> </span><span class="nb">seed</span><span class="w"> </span><span class="n">file</span><span class="w"> </span><span class="n">core</span><span class="o">-</span><span class="n">image</span><span class="o">-</span><span class="n">minimal</span><span class="o">-</span><span class="n">fvp</span><span class="o">-</span><span class="n">base</span><span class="o">-</span><span class="mf">20210610124230.</span><span class="n">rootfs</span><span class="o">.</span><span class="n">wic</span>
<span class="n">core</span><span class="o">-</span><span class="n">image</span><span class="o">-</span><span class="n">minimal</span><span class="o">-</span><span class="n">fvp</span><span class="o">-</span><span class="n">base</span><span class="o">-</span><span class="mf">20210610124230.</span><span class="n">rootfs</span><span class="o">.</span><span class="n">wic</span><span class="o">.</span><span class="w"> </span><span class="n">Target</span><span class="w"> </span><span class="mf">70.5</span><span class="o">%</span><span class="w"> </span><span class="n">complete</span><span class="o">.</span>
<span class="n">downloading</span><span class="w"> </span><span class="n">from</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">buildmachine</span><span class="p">:</span><span class="mi">8080</span><span class="o">/</span><span class="n">core</span><span class="o">-</span><span class="n">image</span><span class="o">-</span><span class="n">sato</span><span class="o">-</span><span class="n">fvp</span><span class="o">-</span><span class="n">base</span><span class="o">-</span><span class="mf">20210610125939.</span><span class="n">rootfs</span><span class="o">.</span><span class="n">wic</span><span class="o">.</span><span class="n">gz</span><span class="p">:</span>
<span class="c1">#################### 100.0% 10071.8 kBps DONE </span>
<span class="n">verifying</span><span class="w"> </span><span class="n">download</span><span class="o">...</span><span class="n">checksum</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="n">OK</span>
<span class="n">used</span><span class="w"> </span><span class="mi">538800128</span><span class="w"> </span><span class="n">local</span><span class="p">,</span><span class="w"> </span><span class="n">fetched</span><span class="w"> </span><span class="mi">70972961</span>
</code></pre></div>
<p>By using the seed file, zsync determined that it already has 70% of the file on disk, and downloaded just the remaining chunks.</p>
<p>For incremental builds the differences can be very small when using the Yocto Project, as thanks to the <a href="https://reproducible-builds.org">reproducible builds</a> effort there are no spurious changes (such as embedded timestamps or non-deterministic compilation) on recompiles.</p>
<p>Now, obviously I don't recommend doing all of this by hand. For Yocto Project users, as of right now there is a <a href="https://patchwork.openembedded.org/patch/179878/">patch queued for <code>meta-openembedded</code></a> adding a recipe for <a href="https://github.com/probonopd/zsync-curl">zsync-curl</a>, and a <a href="https://lists.openembedded.org/g/openembedded-core/message/152831">patch queued for openembedded-core</a> to add <code>zsync</code> and <code>gzsync</code> image conversion types (for <code>IMAGE_FSTYPES</code>, for example <code>wic.gzsync</code>) to generate the metadata automatically. Bring your own HTTP server and you can fetch without further effort.</p>Rewriting Git Commit Messages2018-03-06T17:00:00+00:002018-03-06T17:00:00+00:00Ross Burtontag:www.burtonini.com,2018-03-06:/blog/2018/03/06/rewriting-git-commit-messages/<p>So this week I started submitting a seventy-odd commits long branch where every commit was machine generated (but hand reviewed) with the amazing commit message of "component: refresh patches". Whilst this was easy to automate the message isn't acceptable to merge and I was facing the prospect of copy/pasting …</p><p>So this week I started submitting a seventy-odd commits long branch where every commit was machine generated (but hand reviewed) with the amazing commit message of "component: refresh patches". Whilst this was easy to automate the message isn't acceptable to merge and I was facing the prospect of copy/pasting the same commit message over and over during an interactive rebase. That did not sound like fun. I ended up writing a tiny tool to do this and thought I'd do my annual blog post about it, mainly so I can find it again when I need to do it again next year...</p>
<p>Wise readers will know that Git can rewrite all sorts of things in commits programatically using <code>git-filter-branch</code> and this has a <code>--msg-filter</code> argument which sounds like just what I need. But first a note: <code>git-filter-branch</code> can destroy your branches if you're not careful!</p>
<p><code>git filter-branch --msg-filter</code> has a simple behaviour: give it a command to be executed by the shell, the old commit message is piped in via standard input, and whatever appears on standard output is the new commit message. Sounds simple but in a way it's too simple, as even the example in the documentation has a glaring problem.</p>
<p>Anyway, this should work. I have a commit message in a predictable format (<recipe name>: refresh patches) and a text editor containing a longer message suitable for submission. I could write a bundle of shell/sed/awk to munge from one to the other but I decided to simply glue a few pieces of Python together instead:</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">sys</span><span class="o">,</span> <span class="nn">re</span>
<span class="n">input_re</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="nb">open</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span><span class="o">.</span><span class="n">read</span><span class="p">())</span>
<span class="n">template</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">2</span><span class="p">])</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">original_message</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">match</span> <span class="o">=</span> <span class="n">input_re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">original_message</span><span class="p">)</span>
<span class="k">if</span> <span class="n">match</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="n">template</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="n">match</span><span class="o">.</span><span class="n">groupdict</span><span class="p">()))</span>
<span class="k">else</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="n">original_message</span><span class="p">)</span>
</code></pre></div>
<p>Invoke this with two filenames: a regular expression to match on the input, and a template for the new commit message. If the regular expression matches then any named groups are extracted and passed to the template which is output using the new-style <code>format()</code> operation. If it doesn't match then the input is simply output to preserve commit messages.</p>
<p>This is my input regular expression:</p>
<div class="highlight"><pre><span></span><code>^(?P<recipe>.+): refresh patches
</code></pre></div>
<p>And this is my output template:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="n">recipe</span><span class="p">}:</span><span class="w"> </span><span class="n">refresh</span><span class="w"> </span><span class="n">patches</span>
<span class="n">The</span><span class="w"> </span><span class="n">patch</span><span class="w"> </span><span class="k">tool</span><span class="w"> </span><span class="n">will</span><span class="w"> </span><span class="n">apply</span><span class="w"> </span><span class="n">patches</span><span class="w"> </span><span class="n">by</span><span class="w"> </span><span class="n">default</span><span class="w"> </span><span class="n">with</span><span class="w"> </span><span class="s2">"fuzz"</span><span class="p">,</span><span class="w"> </span><span class="n">which</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">where</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">the</span>
<span class="n">hunk</span><span class="w"> </span><span class="n">context</span><span class="w"> </span><span class="n">isn</span><span class="s1">'t present but what is there is close enough, it will force the</span>
<span class="n">patch</span><span class="w"> </span><span class="ow">in</span><span class="o">.</span>
<span class="n">Whilst</span><span class="w"> </span><span class="n">this</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">useful</span><span class="w"> </span><span class="n">when</span><span class="w"> </span><span class="n">there</span><span class="s1">'s just whitespace changes, when applied to</span>
<span class="n">source</span><span class="w"> </span><span class="n">it</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">possible</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">patch</span><span class="w"> </span><span class="n">applied</span><span class="w"> </span><span class="n">with</span><span class="w"> </span><span class="n">fuzz</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">produce</span><span class="w"> </span><span class="n">broken</span><span class="w"> </span><span class="n">code</span><span class="w"> </span><span class="n">which</span>
<span class="n">still</span><span class="w"> </span><span class="n">compiles</span><span class="w"> </span><span class="p">(</span><span class="n">see</span><span class="w"> </span><span class="c1">#10450). This is obviously bad.</span>
<span class="n">We</span><span class="s1">'d like to eventually have do_patch() rejecting any fuzz on these grounds. For</span>
<span class="n">that</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">realistic</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">existing</span><span class="w"> </span><span class="n">patches</span><span class="w"> </span><span class="n">with</span><span class="w"> </span><span class="n">fuzz</span><span class="w"> </span><span class="n">need</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">rebased</span><span class="w"> </span><span class="ow">and</span>
<span class="n">reviewed</span><span class="o">.</span>
<span class="n">Signed</span><span class="o">-</span><span class="n">off</span><span class="o">-</span><span class="n">by</span><span class="p">:</span><span class="w"> </span><span class="n">Ross</span><span class="w"> </span><span class="n">Burton</span><span class="w"> </span><span class="o"><</span><span class="n">ross</span><span class="o">.</span><span class="n">burton</span><span class="err">@</span><span class="n">intel</span><span class="o">.</span><span class="n">com</span><span class="o">></span>
</code></pre></div>
<p>A quick run through <code>filter-branch</code> and I'm ready to send:</p>
<div class="highlight"><pre><span></span><code>git filter-branch --msg-filter 'rewriter.py input output' origin/master...HEAD
</code></pre></div>Identifying concurrent tasks in Bitbake logs2017-06-20T15:24:00+01:002017-06-20T15:24:00+01:00Ross Burtontag:www.burtonini.com,2017-06-20:/blog/2017/06/20/identifying-concurrent-tasks-in-bitbake-logs/<p>One fun problem in massively parallel OpenEmbedded builds is when tasks have bad dependencies or just bugs and you can end up with failures due to races on disk.</p>
<p>One example of this happened last week when an integration branch was being tested and one of the builds failed with …</p><p>One fun problem in massively parallel OpenEmbedded builds is when tasks have bad dependencies or just bugs and you can end up with failures due to races on disk.</p>
<p>One example of this happened last week when an integration branch was being tested and one of the builds failed with <code>tar error: file changed as we read it</code> whilst it was generating the images. This means that the root filesystem was being altered whilst tar was reading it, so we've a parallelism problem. There's only a limited number of tasks that could be having this effect here so searching the log isn't too difficult, but as they say: why do something by hand when you can <a href="https://raw.githubusercontent.com/rossburton/ross-tools/master/findfails.py">write a script to do it for you</a>.</p>
<p><a href="https://raw.githubusercontent.com/rossburton/ross-tools/master/findfails.py"><code>findfails</code></a> is a script that will parse a Bitbake log and maintain the set of currently active tasks, so when it finds a task that fails it can tell you what other tasks are also running:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>findfails<span class="w"> </span>log
Task<span class="w"> </span>core-image-sato-dev-1.0-r0:do_image_tar<span class="w"> </span>failed
Active<span class="w"> </span>tasks<span class="w"> </span>are:
<span class="w"> </span>core-image-sato-sdk-ptest-1.0-r0:do_rootfs
<span class="w"> </span>core-image-sato-dev-1.0-r0:do_image_wic
<span class="w"> </span>core-image-sato-dev-1.0-r0:do_image_jffs2
<span class="w"> </span>core-image-sato-dev-1.0-r0:do_image_tar
<span class="w"> </span>core-image-sato-sdk-1.0-r0:do_rootfs<span class="sb">```</span>
</code></pre></div>
<p>We knew that there were changes to <code>do_image_wic</code> in that branch, so it was easy to identify and drop the patch that was incorrectly writing to the rootfs source directory. Sorted!</p>Dynamic source checksums in OpenEmbedded2017-06-13T14:43:00+01:002017-06-13T14:43:00+01:00Ross Burtontag:www.burtonini.com,2017-06-13:/blog/2017/06/13/dynamic-source-checksums/<p>Today we were cleaning up some old bugs in the <a href="http://www.yoctoproject.org/">Yocto Project</a> bugzilla and came across a bug which was asking for the ability to specify a remote URL for the source tarball checksums (<code>SRC_URI[md5sum]</code> and/or <code>SRC_URI[sha256um]</code>). We require a checksum for tarballs for two reasons:</p>
<ol>
<li>Download …</li></ol><p>Today we were cleaning up some old bugs in the <a href="http://www.yoctoproject.org/">Yocto Project</a> bugzilla and came across a bug which was asking for the ability to specify a remote URL for the source tarball checksums (<code>SRC_URI[md5sum]</code> and/or <code>SRC_URI[sha256um]</code>). We require a checksum for tarballs for two reasons:</p>
<ol>
<li>Download integrity. We want to be sure that the download wasn't corrupted in some way, such as truncation or bad encoding.</li>
<li>Security. We want to be sure that the tarball hasn't changed over time, be it the maintainer regenerating the tarball for an old release but with different content (this happens more than you'd expect, with non-trivial changes too), or alternatively a malicious attack on the file which now contains malware (such as the <a href="https://blog.malwarebytes.com/threat-analysis/mac-threat-analysis/2017/05/handbrake-hacked-to-drop-new-variant-of-proton-malware/">Handbrake hack in May</a>).</li>
</ol>
<p>The rationale for reading remote URLs for checksums was that for files that are changing frequently it would be easier to upgrade the recipe if the checksums didn't need to be altered too. For some situations I can see this argument, but I don't want to encourage practices that nullify the security checksums. For this reason I rejected the bug but thanks to the power of Bitbake I did provide a working example of how to do this in your recipe.</p>
<p>The trick is to observe that the only time the <code>SRC_URI[md5sum]</code> is read is during <code>do_fetch</code>. By adding a new function to <code>do_fetch[prefuncs]</code> (the list of functions that will be executed before <code>do_fetch</code> is executed) we can download the checksums and write the variable just before the fetcher needs it. Here is a partial example that works for GNOME-style checksums, where each upload generates <code>foo-1.2.tar.bz2</code>, <code>foo-1.2.tar.xz</code>, <code>foo-1.2.sha256sum</code>, and <code>foo-1.2.md5sum</code>. To keep it interesting the checksum files contain the sums for both compression types, so we need to iterate through the file to find the right line:</p>
<div class="highlight"><pre><span></span><code><span class="n">SRC_URI</span> <span class="o">=</span> <span class="s2">"https://download.gnome.org/sources/glib/2.52/glib-2.52.2.tar.xz"</span>
<span class="n">SHASUM_URI</span> <span class="o">=</span> <span class="s2">"https://download.gnome.org/sources/glib/2.52/glib-2.52.2.sha256sum"</span>
<span class="n">do_fetch</span><span class="p">[</span><span class="n">prefuncs</span><span class="p">]</span> <span class="o">+=</span> <span class="s2">"fetch_checksums"</span>
<span class="n">python</span> <span class="n">fetch_checksums</span><span class="p">()</span> <span class="p">{</span>
<span class="kn">import</span> <span class="nn">urllib</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">d</span><span class="o">.</span><span class="n">getVar</span><span class="p">(</span><span class="s2">"SHASUM_URI"</span><span class="p">)):</span>
<span class="p">(</span><span class="n">sha</span><span class="p">,</span> <span class="n">filename</span><span class="p">)</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"ascii"</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
<span class="k">if</span> <span class="n">filename</span> <span class="o">==</span> <span class="s2">"glib-2.52.2.tar.xz"</span><span class="p">:</span>
<span class="n">d</span><span class="o">.</span><span class="n">setVarFlag</span><span class="p">(</span><span class="s2">"SRC_URI"</span><span class="p">,</span> <span class="s2">"sha256sum"</span><span class="p">,</span> <span class="n">sha</span><span class="p">)</span>
<span class="k">return</span>
<span class="n">bb</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Could not find remote checksum"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>Note that as <code>fetch_checksums</code> is a pre-function for <code>do_fetch</code> it is only executed just before <code>do_fetch</code> and not at any other time, so this doesn't impose any delays on builds that don't need to fetch.</p>
<p>If I were taking this beyond a proof of concept and making it into a general-purpose class there's a number of changes I would want to make:</p>
<ol>
<li>Use the proxies when calling <code>urlopen()</code></li>
<li>Extract the filename to search for from the <code>SRC_URI</code></li>
<li>Generate the checksum URL from the <code>SRC_URI</code></li>
</ol>
<p>I'll leave those as an exercise to the reader though. Patches welcome!</p>So long Wordpress, thanks for all the exploits2016-08-26T23:10:00+01:002016-08-26T23:10:00+01:00Ross Burtontag:www.burtonini.com,2016-08-26:/blog/2016/08/26/so-long-wordpress/<p>I've been meaning to move my incredibly occasional blog away from Wordpress for a long time, considering that I rarely use my blog and it's a massive attack surface. But there's always more important things to do, so I never did.</p>
<p>Then in the space of ten days I received …</p><p>I've been meaning to move my incredibly occasional blog away from Wordpress for a long time, considering that I rarely use my blog and it's a massive attack surface. But there's always more important things to do, so I never did.</p>
<p>Then in the space of ten days I received two messages from my web host, one that they'd discovered a spam bot running on my account, and after that was cleared and passwords reset <em>another</em> that they discovered a password cracker.</p>
<p>Clearly I needed to do something. A little research led me to <a href="http://www.getpelican.com">Pelican</a>, which ticks my "programming language I can read" (Python), "maintained", and "actually works" boxes. A few evenings of fiddling later and I just deleted both Wordpress and Pyblosxom from my host, so hopefully that's the end of the exploits.</p>
<p>No doubt there's some links that are now dead, all the comments have disappeared, and the theme needs a little tweaking still, but that is all relatively minor stuff. I promise to blog more, too.</p>Cycling Dad Nirvana Approaches2015-04-19T21:44:00+01:002015-04-19T21:44:00+01:00Ross Burtontag:www.burtonini.com,2015-04-19:/blog/2015/04/19/cycling-dad-nirvana-approaches/<p>Last week Alex wanted to go for a bike ride so we had a play on the
local pump track and some of the cheeky trails hidden away nearby. This
was his first ride off the pavements so I was cautious but <a href="https://vimeo.com/124923068">much fun was
had</a> by Alex and he's …</p><p>Last week Alex wanted to go for a bike ride so we had a play on the
local pump track and some of the cheeky trails hidden away nearby. This
was his first ride off the pavements so I was cautious but <a href="https://vimeo.com/124923068">much fun was
had</a> by Alex and he's spent most of the
last week talking about the ride and in particular the pump track.
There's a pretty good one at Thetford Forest now so now that Spring has
(mostly) sprung we decided to have a family day out and give Isla some
proper practise at riding her new bike.</p>
<p><div class="">
<a href="https://www.flickr.com/photos/rossburton/16995308217/">
<img class="center-block" src="//farm8.static.flickr.com/7680/16995308217_bb4f7a3867_z.jpg" alt="Family ride at Thetford Forest"
/>
</a>
<div class="caption">
<p>Family ride at Thetford Forest</p>
</div>
</div></p>
<p>For the start of the ride it was me and Alex riding ahead with Vicky
riding alongside Isla whilst she practised the hard bit of stopping and
starting. It didn't take long before we heard a loud "COMING THROUGH!"
and Isla flew by. A few kilometres down the Shepherd trail we decided to
head (badly, I can't recall the Shepherd route at all) before little
legs tired and find the pump track. There were a few tumbles, Alex was
getting tired and there's a tight berm with a very loose straight line.
Isla of course had the confidence of a bold little sister and wanted a
go, which led to a great double face-plant when I was running alongside
guiding her around. Nothing a bit of Savlon won't solve though, and ice
cream made it all better!</p>
<p>All in all, a good day: Alex had a good ride and fun on the pump track,
and Isla is massively more confident on her bike, especially when the
path isn't new-pavement-smooth. Not panicking when the path is a bit
"bumpy lumpy" is an important skill when riding alongside traffic!</p>UEFI Validation2014-05-19T10:07:00+01:002014-05-19T10:07:00+01:00Ross Burtontag:www.burtonini.com,2014-05-19:/blog/2014/05/19/uefi-validation/<p>The <a href="https://01.org/linux-uefi-validation">Linux UEFI Validation Project</a> was announced recently:</p>
<blockquote>
<p>Bringing together multiple separate upstream test suites into a
cohesive and easy-to-use product with a unified reporting framework,
LUV validates UEFI firmware at critical levels of the Linux software
stack and boot phases.</p>
<p>LUV also provides tests in areas not previously available …</p></blockquote><p>The <a href="https://01.org/linux-uefi-validation">Linux UEFI Validation Project</a> was announced recently:</p>
<blockquote>
<p>Bringing together multiple separate upstream test suites into a
cohesive and easy-to-use product with a unified reporting framework,
LUV validates UEFI firmware at critical levels of the Linux software
stack and boot phases.</p>
<p>LUV also provides tests in areas not previously available, such as the
interaction between the bootloader, Linux kernel and firmware.
Integrated into one Linux distribution, firmware can now be tested
under conditions that are closer to real scenarios of operation. The
result: fewer firmware issues disrupting the operating system.</p>
</blockquote>
<p>Of course that "one Linux distribution" is built using the <a href="https://www.yoctoproject.org/">Yocto
Project</a>, so it's trivial to grab the
source and patch/extend it, or rebuild it for different processors if
for example you want to validate UEFI on
<a href="https://wiki.linaro.org/ARM/UEFI">ARM</a> or a <a href="http://en.wikipedia.org/wiki/Intel_Quark">new CPU that mainstream
distros don't fully support</a>.</p>Reproducible builds and GPL compliance2014-05-15T23:27:00+01:002014-05-15T23:27:00+01:00Ross Burtontag:www.burtonini.com,2014-05-15:/blog/2014/05/15/reproducible-builds-and-gpl-compliance/<p>LWN has a good <a href="http://lwn.net/Articles/598371/">article on GPL
compliance</a> (if you're not a subscriber
you'll have to wait) that has an interesting quote:</p>
<blockquote>
<p>Developers, and embedded developers in particular, can help stop these
violations. When you get code from a supplier, ensure that you can
build it, he said, because someone …</p></blockquote><p>LWN has a good <a href="http://lwn.net/Articles/598371/">article on GPL
compliance</a> (if you're not a subscriber
you'll have to wait) that has an interesting quote:</p>
<blockquote>
<p>Developers, and embedded developers in particular, can help stop these
violations. When you get code from a supplier, ensure that you can
build it, he said, because someone will eventually ask. Consider using
the Yocto Project, as Beth Flanagan has been adding a number of
features to Yocto to help with GPL compliance. Having reproducible
builds is simply good engineering practice—if you can't reproduce your
build, you have a problem.</p>
</blockquote>
<p>This has always been one of the key points that we emphasis when
explaining why you should use the Yocto Project for your next product.
If you're shipping a product that is built using fifty open source
projects then ensuring that you can redistribute all the original
sources, and the patches that you've applied, and the configure options
that you've used, and any tweaks to go from a directory of binaries to a
bootable image isn't something you can knock up in an afternoon when you
get a letter from the <a href="http://sfconservancy.org/">SFC</a>. Fingers crossed
you didn't accidentally use some GPLv3 code when that is considered
toxic.</p>
<p>Beth is awesome and has worked with others in the Yocto community to
ensure all of this is covered. Yocto can produce license manifests,
upstream sources + patches archives, verify GPLv3 code isn't
distributed, and more. All the work that is terribly boring at the
beginning when you have a great idea and are full of enthusiasm (and
Club-Mate), but by the time you're shipping is often nigh on impossible.
Dave built the kernel on his machine but the disk with the right source
tree on died, and Sarah left without telling anyone else the right flags
to make libhadjaha actually link... it'll be fine, right?</p>