Moving firmware to userland

[Jan Sucan]

Hello Adrian,

thank you for the tips.

Version information could be included in these ways:
- Version number in the header. This would express version of all
   records/sections in the file.
- Version number included in the ID value. This would express version
   for only the record. So there will be possible to have records of
   multiple versions in the same file.

jan


On Tue, 25 Jul 2017, Adrian Chadd wrote:

> hi,
>
> Since I've been thinking of doing this in freebsd, I highly suggest
> the following:
>
> * version numbers in the header - so you know what version your
> firmware files are
> * use a TLV format so you can add and find arbitrary sectoins, kinda
> like elf binaries but not elf binaries.
>
>
>
> -adrian
>
>
> On 23 July 2017 at 03:00, Jan Sucan <sucanjan at fit.cvut.cz> wrote:
>> Hello,
>>
>> this is the proposal for the new firmware file format.
>>
>> ---- Firmware file format ----
>>
>> Data structure used is a singly linked list. Operations on it have a
>> linear time complexity (worst-case), but because there is only a few
>> records it is sufficient.
>>
>> Conditions for ordering of the records:
>> - checksum record is the first
>> - firmware image data is the last
>>
>> Checksum and image data are the only mandatory records.
>>
>> Start of the linked list is at the end of the file (and it continues
>> "backwards") so the file can be resized more easily when modifying
>> records (firmware attributes).
>>
>> Each record has an ID (1 byte) which describes the meaning of it and
>> its allowed values. ID is followed by the length of attribute data in
>> bytes (4 bytes, little-endian). The next record starts immediately
>> after data of the current record.
>>
>> Each record is read-write for a user except for the two read-only
>> records:
>> - checksum
>> - firmware image data
>>
>> ---- Records ----
>>
>> - Checksum: string (checksum will cover everything after this record)
>> - License text: string (file path or complete text of the license, no
>>   license ack is needed if not defined)
>> - Firmware version: integer number
>> - Firmware image endianness: boolean (little-endian, big-endian)
>> - Firmware image word size: 1,2,4 or 8 (used for endianness)
>> - Firmware image data: array
>>
>> ---- Operations with firmware files ----
>>
>> - Initialize raw firmware file: create linked list structure and
>>   checksum and firmware image data records
>> - Uninitialize firmware file: remove linked list structure and keep
>>   only the original raw firmware data
>> - Check firmware using the checksum
>> - Set record's value: only for read-write records (this also creates
>>   record which is not defined yet)
>> - Get record's value: for read-only and read-write records
>> - Remove record: only for read-write records
>>
>>
>> jan
>>
>>
>>
>> On Thu, 20 Jul 2017, Sepherosa Ziehau wrote:
>>
>>> Hi,
>>>
>>> On Wed, Jul 19, 2017 at 4:27 PM, Jan Sucan <sucanjan at fit.cvut.cz> wrote:
>>>>
>>>> That's good and interesting idea. Attaching the config to the image
>>>> could unify the way in which config info is saved for both built-in
>>>> and non-built-in images, and also unify a code for parsing the config
>>>> info in the kernel firmware subsystem. Possibilities for firmware file
>>>> formats:
>>>>
>>>> - uuencoded
>>>>   Pros: Config can be added, modified and removed by a text editor
>>>>   Cons: .uu files are about 35% larger, need to implement uudecoding in
>>>>         the kernel, config correctness checked at firmware loading time
>>>>
>>>> - binary
>>>>   Pros: firmware data size, user-interface of the new tool can be
>>>>         better suited for given purpose than a text editor (which is too
>>>>         general), config correctness checked at config modifying time
>>>>   Cons: need to create new tool for handling attached info (something
>>>>         like kenv(1) for firmware)
>>>>
>>>> For binary firmware it will be necessary to also add some checksum
>>>> data in case when config is not attached and firmware data is also
>>>> valid config. Checksum could be computed only from firmware data or it
>>>> can cover the config data too.
>>>>
>>>> I think, that the second file format would be better. I can try to
>>>> propose a format for the attached config and post it here for a
>>>> review.
>>>
>>>
>>> Yeah, sure, go ahead :)
>>>
>>> Thank you,
>>> sephe
>>>
>>>>
>>>>
>>>>
>>>> On Tue, 18 Jul 2017, Sepherosa Ziehau wrote:
>>>>
>>>>> I'd choose plain text config per-image.  BTW, is it possible to attach
>>>>> the config at the beginning/end of the image?  Assuming we can force
>>>>> some image file creation rules.
>>>>>
>>>>> On Tue, Jul 11, 2017 at 8:20 PM, Ján Sučan <sucanjan at fit.cvut.cz> wrote:
>>>>>>
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I would like to introduce a few ideas about the new firmware subsystem.
>>>>>> I
>>>>>> assume that it should not require adding a new system tools or
>>>>>> modifying
>>>>>> boot process.
>>>>>>
>>>>>> Simplification is the first. It would be good to remove parent-child
>>>>>> relationship and corresponding functionality. It would significantly
>>>>>> simplify firmware handling. Its only practical use is when there are
>>>>>> multiple images in one loadable kernel module. The module can be
>>>>>> unloaded
>>>>>> when all children are not in use. Usage of the children images is
>>>>>> tracked
>>>>>> through the counter for the parent image. If images will not be placed
>>>>>> inside loadable kernel modules, the parent-child mechanism won't have
>>>>>> any
>>>>>> practical meaning. I think, currently the mechanism is not used
>>>>>> anywhere
>>>>>> in
>>>>>> the DragonFly system and if it was, it could be easily replaced by
>>>>>> putting
>>>>>> every child image in its own module without modifying drivers.
>>>>>>
>>>>>> There are two use cases according to who will provide firmware images
>>>>>> to
>>>>>> the
>>>>>> system:
>>>>>>
>>>>>> - developers of DragonFly BSD (they can modify and rebuild the system)
>>>>>> - third-parties (they should not be required to modify and rebuild the
>>>>>> system)
>>>>>>
>>>>>> Providing a new non-built-in firmware should not require:
>>>>>>
>>>>>> a) system rebuild
>>>>>> b) system reboot
>>>>>> c) loading a kernel module
>>>>>>
>>>>>> All firmware images needs to have some information attached (at least,
>>>>>> if
>>>>>> ack with a license is needed) which should be
>>>>>>
>>>>>> d) stored persistently.
>>>>>>
>>>>>> The question is where to save the information for non-built-in images.
>>>>>> For
>>>>>> built-in images the info is saved in the kernel. Some possibilities
>>>>>> are:
>>>>>>
>>>>>> - A plain text configuration file per image.
>>>>>>   This satisfies requirements a), b), c) and d). Disadvantage is
>>>>>> parsing
>>>>>> the
>>>>>> text content.
>>>>>>
>>>>>> - Kernel environment variables.
>>>>>>   The problem with this is that they are initialized:
>>>>>>     - by some kernel code (this violates a), b), c) )
>>>>>>     - manually from userland by kenv command or kenv() libc function
>>>>>> (violates d) )
>>>>>>     - in the loader.conf (violates b) since the content is processed at
>>>>>> boot
>>>>>> time)
>>>>>>   Advantage is that the parsing is done by the kernel.
>>>>>>
>>>>>> There would be two firmware sources: kernel and filesystem. In case of
>>>>>> the
>>>>>> same image names, user could have a choice by setting a kernel
>>>>>> environment
>>>>>> variable, firmware from which source has higher priority and will be
>>>>>> provided to consumer.
>>>>>>
>>>>>>
>>>>>> I will be happy for a comments and ideas
>>>>>> jan
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Tomorrow Will Never Die
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Tomorrow Will Never Die
>>>
>>
>
>