Moving firmware to userland

[Sepherosa Ziehau]

Looks good to me.  I'd leave some spare fields in the in-image record
head (zero filled), for future extension of the in-image record head.

On Sun, Jul 23, 2017 at 6:00 PM, Jan Sucan <sucanjan at fit.cvut.cz> wrote:
> Hello,
>
> this is the proposal for the new firmware file format.
>
> ---- Firmware file format ----
>
> Data structure used is a singly linked list. Operations on it have a
> linear time complexity (worst-case), but because there is only a few
> records it is sufficient.
>
> Conditions for ordering of the records:
> - checksum record is the first
> - firmware image data is the last
>
> Checksum and image data are the only mandatory records.
>
> Start of the linked list is at the end of the file (and it continues
> "backwards") so the file can be resized more easily when modifying
> records (firmware attributes).
>
> Each record has an ID (1 byte) which describes the meaning of it and
> its allowed values. ID is followed by the length of attribute data in
> bytes (4 bytes, little-endian). The next record starts immediately
> after data of the current record.
>
> Each record is read-write for a user except for the two read-only
> records:
> - checksum
> - firmware image data
>
> ---- Records ----
>
> - Checksum: string (checksum will cover everything after this record)
> - License text: string (file path or complete text of the license, no
>   license ack is needed if not defined)
> - Firmware version: integer number
> - Firmware image endianness: boolean (little-endian, big-endian)
> - Firmware image word size: 1,2,4 or 8 (used for endianness)
> - Firmware image data: array
>
> ---- Operations with firmware files ----
>
> - Initialize raw firmware file: create linked list structure and
>   checksum and firmware image data records
> - Uninitialize firmware file: remove linked list structure and keep
>   only the original raw firmware data
> - Check firmware using the checksum
> - Set record's value: only for read-write records (this also creates
>   record which is not defined yet)
> - Get record's value: for read-only and read-write records
> - Remove record: only for read-write records
>
>
> jan
>
>
>
> On Thu, 20 Jul 2017, Sepherosa Ziehau wrote:
>
>> Hi,
>>
>> On Wed, Jul 19, 2017 at 4:27 PM, Jan Sucan <sucanjan at fit.cvut.cz> wrote:
>>>
>>> That's good and interesting idea. Attaching the config to the image
>>> could unify the way in which config info is saved for both built-in
>>> and non-built-in images, and also unify a code for parsing the config
>>> info in the kernel firmware subsystem. Possibilities for firmware file
>>> formats:
>>>
>>> - uuencoded
>>>   Pros: Config can be added, modified and removed by a text editor
>>>   Cons: .uu files are about 35% larger, need to implement uudecoding in
>>>         the kernel, config correctness checked at firmware loading time
>>>
>>> - binary
>>>   Pros: firmware data size, user-interface of the new tool can be
>>>         better suited for given purpose than a text editor (which is too
>>>         general), config correctness checked at config modifying time
>>>   Cons: need to create new tool for handling attached info (something
>>>         like kenv(1) for firmware)
>>>
>>> For binary firmware it will be necessary to also add some checksum
>>> data in case when config is not attached and firmware data is also
>>> valid config. Checksum could be computed only from firmware data or it
>>> can cover the config data too.
>>>
>>> I think, that the second file format would be better. I can try to
>>> propose a format for the attached config and post it here for a
>>> review.
>>
>>
>> Yeah, sure, go ahead :)
>>
>> Thank you,
>> sephe
>>
>>>
>>>
>>>
>>> On Tue, 18 Jul 2017, Sepherosa Ziehau wrote:
>>>
>>>> I'd choose plain text config per-image.  BTW, is it possible to attach
>>>> the config at the beginning/end of the image?  Assuming we can force
>>>> some image file creation rules.
>>>>
>>>> On Tue, Jul 11, 2017 at 8:20 PM, Ján Sučan <sucanjan at fit.cvut.cz> wrote:
>>>>>
>>>>>
>>>>> Hello,
>>>>>
>>>>> I would like to introduce a few ideas about the new firmware subsystem.
>>>>> I
>>>>> assume that it should not require adding a new system tools or
>>>>> modifying
>>>>> boot process.
>>>>>
>>>>> Simplification is the first. It would be good to remove parent-child
>>>>> relationship and corresponding functionality. It would significantly
>>>>> simplify firmware handling. Its only practical use is when there are
>>>>> multiple images in one loadable kernel module. The module can be
>>>>> unloaded
>>>>> when all children are not in use. Usage of the children images is
>>>>> tracked
>>>>> through the counter for the parent image. If images will not be placed
>>>>> inside loadable kernel modules, the parent-child mechanism won't have
>>>>> any
>>>>> practical meaning. I think, currently the mechanism is not used
>>>>> anywhere
>>>>> in
>>>>> the DragonFly system and if it was, it could be easily replaced by
>>>>> putting
>>>>> every child image in its own module without modifying drivers.
>>>>>
>>>>> There are two use cases according to who will provide firmware images
>>>>> to
>>>>> the
>>>>> system:
>>>>>
>>>>> - developers of DragonFly BSD (they can modify and rebuild the system)
>>>>> - third-parties (they should not be required to modify and rebuild the
>>>>> system)
>>>>>
>>>>> Providing a new non-built-in firmware should not require:
>>>>>
>>>>> a) system rebuild
>>>>> b) system reboot
>>>>> c) loading a kernel module
>>>>>
>>>>> All firmware images needs to have some information attached (at least,
>>>>> if
>>>>> ack with a license is needed) which should be
>>>>>
>>>>> d) stored persistently.
>>>>>
>>>>> The question is where to save the information for non-built-in images.
>>>>> For
>>>>> built-in images the info is saved in the kernel. Some possibilities
>>>>> are:
>>>>>
>>>>> - A plain text configuration file per image.
>>>>>   This satisfies requirements a), b), c) and d). Disadvantage is
>>>>> parsing
>>>>> the
>>>>> text content.
>>>>>
>>>>> - Kernel environment variables.
>>>>>   The problem with this is that they are initialized:
>>>>>     - by some kernel code (this violates a), b), c) )
>>>>>     - manually from userland by kenv command or kenv() libc function
>>>>> (violates d) )
>>>>>     - in the loader.conf (violates b) since the content is processed at
>>>>> boot
>>>>> time)
>>>>>   Advantage is that the parsing is done by the kernel.
>>>>>
>>>>> There would be two firmware sources: kernel and filesystem. In case of
>>>>> the
>>>>> same image names, user could have a choice by setting a kernel
>>>>> environment
>>>>> variable, firmware from which source has higher priority and will be
>>>>> provided to consumer.
>>>>>
>>>>>
>>>>> I will be happy for a comments and ideas
>>>>> jan
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Tomorrow Will Never Die
>>>>
>>>
>>
>>
>>
>> --
>> Tomorrow Will Never Die
>>
>



-- 
Tomorrow Will Never Die