Netfilter and Multicast

Netfilter can by default not track replies to multicast(/broadcast for IPv4) messages. Thus they are, unless they are explicitely allowed through a stateless rule, not matched by a rule using the conntrack match module. To work around this limitation, I developed a equivalent workaround using ipset and iptables. Theoretically the same can be done with nftables easily. But it has to be done manually. Unless that is done, multicast and broadcast communication is severely limited.

ipset definitions for ipset.conf:
create track-multicast4 hash:net,port family inet timeout 15
create track-multicast6 hash:net,port family inet6 timeout 15

iptables rules:
-A INPUT -m set --match-set track-multicast4 dst,dst -j ACCEPT
-A OUTPUT -m addrtype --dst-type BROADCAST -j SET --add-set track-multicast4 src,src
-A OUTPUT -m addrtype --dst-type MULTICAST -j SET --add-set track-multicast4 src,src

ip6tables rules:
-A INPUT -m set --match-set track-multicast6 dst,dst -j ACCEPT
-A OUTPUT -m addrtype --dst-type MULTICAST -j SET --add-set track-multicast6 src,src

Getting IRC help: what information do I need to provide?

The necessary information to enable others to help you will of course vary with the problem for which you are seeking help. But here are some minimum guidelines, for users of the #netfilter channel on freenode.net IRC.

First off, it’s always a good idea to put a summary of the problem and goal in your paste. Think of the convenience of the people you are hoping will help you. They are not as deeply involved in your issue as you are. They have jobs and lives and responsibilities of their own, outside of IRC. So start off with this.

For iptables (iptables-legacy) users, you’ll usually need to provide your complete ruleset including packet and byte counters:

iptables-save -c

For nftables users, similarly, you should show your entire ruleset:

nft list ruleset

If the issue involves routing, and believe it or not, most of them do, it would not hurt to include this:

ip addr list ; ip route list ; ip rule list

There are lots of pastebin choices, but I’ll just mention a couple that I know are adequate. My favorite tends to be ix.io, but only if you go to the extra trouble to concatenate all your information into a single paste. I’m not really a fan of pastebin.com, but their “raw” links are good. I really dislike side-scrolling for long lines, text boxes with up/down and left/right scrollbars.

Finally, remember, you’re not entitled to free professional assistance; if you get it, accept it with gratitude. And if you do not, well, that’s too bad, but if you are gracious and dignified, maybe someone will be available and willing to help you later.

Direct firmware load for $something_mc.bin failed with error -2

-2 means that the file was not found. Install linux-firmware or whatever the name for the linux firmware package is of your distro. Install it and reboot. Then that error should be gone. If it’s not, double check if $something_mc.bin exists and check where to get it. It might just not be packaged by the maintainers of the package.

for indexers:
Direct firmware load for amdgpu/polaris10_mc.bin failed with error -2

Working with SetupDiGetDriverInfoDetailA

When using the Windows setupapi function SetupDiGetDriverInfoDetailA, it can set the thread local error variable to ERROR_INSUFFICIENT_BUFFER or ERROR_INVALID_USER_BUFFER.

The two mean basically the same thing: Your buffer is too small. The SP_DRVINFO_DETAIL_DATA_A struct has a dynamically sized tail that contains the Hardware IDs. ERROR_INVALID_USER_BUFFER means that that dynamically sized area is too small.

Continue reading “Working with SetupDiGetDriverInfoDetailA”

foldingathome in docker and AMD GPUs

In order to run code on AMD GPUs, the AMD provided opencl libraries and other files need to be available in the container. If they are not already present in the docker image, they need to be installed after the fact, besides the fact that the devices need to be made available to the container via the –device parameter.

The following components need to be present in a container to make use of opencl:

  • GPU device card* and associated renderD* device file
  • opencl libraries for the card

The card* and renderD* devices are in /dev/dri/.

The render device associated with a specific card is the card’s number + 128. So renderD128 belongs to card0, renderD129 belongs to card1, and so on. The renderD device and the card device that belong together need to be made available to the same container. You can see the mapping of physcial devices to render and card devices in /dev/dri/by-path/.

Example for the linuxserver/foldingathome image

In the following example, the specific steps to make opencl usable in the docker image linuxserver/foldingathome are shown. The example will store the foldingathome’s software configuration in the host directory /opt/foldingathome and forward card0 and renderD128 into the docker container. The card is a AMD RX580.

Create the docker container: docker container create -c 10 --blkio-weight 10 --name foldingathome -ti --volume /opt/foldingathome:/config --device /dev/dri/card0 --device /dev/dri/renderD128 linuxserver/foldingathome

You can amend the command as required. The important parts are: --volume /opt/foldingathome:/config --device /dev/dri/card0 --device /dev/dri/renderD128

Download the amdgpu driver archive: wget --referer https://www.amd.com/en/support/kb/release-notes/rn-amdgpu-unified-linux-20-20 -N https://drivers.amd.com/drivers/linux/amdgpu-pro-20.20-1089974-ubuntu-18.04.tar.xz
The URL to the drivers can be changed for newer driver versions. The source parameter in the Arch Linux AUR’s opencl-amd package’ PKGBUILD ususally shows the latest one.

Start the container: docker start foldingathome

Copy the driver archive into it: docker cp amdgpu-pro-20.20-1089974-ubuntu-18.04.tar.xz foldingathome:/tmp/

Get a shell in it: docker exec -ti foldingathome bash

Update the package index and install the xz-utils package for extracting the archive: apt update && apt install xz-utils

Extract and install the driver (it will fail because of unmet dependencies, but the opencl driver installs fine): cd /tmp && tar xf amdgpu-pro-20.20-1089974-ubuntu-18.04.tar.xz && cd amdgpu-pro-20.20-1089974-ubuntu-18.04.tar.xz && ./amdgpu-pro-install

Cleanup the files we don’t need: rm -rf /tmp/amdgpu-pro-20.20-1089974-ubuntu-18.04.tar.xz

Install the package we actually want. The install takes some time and unneeded dkms modules are installed, but that doesn’t matter: apt install opencl-amdgpu-pro

Install clinfo to check if the device is usable: apt install clinfo

Check the clinfo output to see if the device is usable. Looks like that if it works: clinfo

Number of platforms                               1                                                                              [188/9922]
  Platform Name                                   AMD Accelerated Parallel Processing
  Platform Vendor                                 Advanced Micro Devices, Inc.
  Platform Version                                OpenCL 2.1 AMD-APP (3075.10)
  Platform Profile                                FULL_PROFILE                                                                             
  Platform Extensions                             cl_khr_icd cl_amd_event_callback cl_amd_offline_devices 
  Platform Host timer resolution                  1ns 
  Platform Extensions function suffix             AMD           
                                                                     
  Platform Name                                   AMD Accelerated Parallel Processing
Number of devices                                 2   
  Device Name                                     Ellesmere
  Device Vendor                                   Advanced Micro Devices, Inc.                                                             
  Device Vendor ID                                0x1002    
  Device Version                                  OpenCL 1.2 AMD-APP (3075.10)                                                             
  Driver Version                                  3075.10                                                                                  
  Device OpenCL C Version                         OpenCL C 1.2                                                                             
  Device Type                                     GPU                                                                                      
  Device Board Name (AMD)                         Radeon RX 580 Series                                                                     
  Device Topology (AMD)                           PCI-E, 0a:00.0                                                                           
  Device Profile                                  FULL_PROFILE                                                                             
  Device Available                                Yes           
  Compiler Available                              Yes               
  Linker Available                                Yes                                                                                        Max compute units                               36                                                                                         SIMD per compute unit (AMD)                     4                                                                                          SIMD width (AMD)                                16                                                                                       
  SIMD instruction width (AMD)                    1            
  Max clock frequency                             1365MHz      
  Graphics IP (AMD)                               8.0        
  Device Partition                                (core)     
    Max number of sub-devices                     36                                                                                       
    Supported partition types                     None         
  Max work item dimensions                        3                 
  Max work item sizes                             1024x1024x1024                                                                           
  Max work group size                             256      
  Preferred work group size (AMD)                 256                                                                                      
  Max work group size (AMD)                       1024          
  Preferred work group size multiple              64                                                                                       
  Wavefront width (AMD)                           64                                                                                       
  Preferred / native vector sizes                          
    char                                                 4 / 4                                                                   [147/9922]
    short                                                2 / 2                                                                             
    int                                                  1 / 1        
    long                                                 1 / 1                                                                             
    half                                                 1 / 1        (cl_khr_fp16)
    float                                                1 / 1        
    double                                               1 / 1        (cl_khr_fp64)                                                        
  Half-precision Floating-point support           (cl_khr_fp16)
    Denormals                                     No
    Infinity and NANs                             No
    Round to nearest                              No
    Round to zero                                 No
    Round to infinity                             No
    IEEE754-2008 fused multiply-add               No
    Support is emulated in software               No
  Single-precision Floating-point support         (core)
    Denormals                                     No
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  Yes
  Double-precision Floating-point support         (cl_khr_fp64)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
  Address bits                                    64, Little-Endian
  Global memory size                              8479080448 (7.897GiB)
  Global free memory (AMD)                        8260768 (7.878GiB)
  Global memory channels (AMD)                    8
  Global memory banks per channel (AMD)           16
  Global memory bank width (AMD)                  256 bytes
  Error Correction support                        No
  Max memory allocation                           4244635648 (3.953GiB)
  Unified memory for Host and Device              No
  Minimum alignment for any data type             128 bytes
  Alignment of base address                       2048 bits (256 bytes)
  Global Memory cache type                        Read/Write
  Global Memory cache size                        16384 (16KiB)
  Global Memory cache line size                   64 bytes
  Image support                                   Yes
    Max number of samplers per kernel             16
    Max size for 1D images from buffer            134217728 pixels
    Max 1D or 2D image array size                 2048 images
    Base address alignment for 2D image buffers   256 bytes
    Pitch alignment for 2D image buffers          256 pixels
    Max 2D image size                             16384x16384 pixels
    Max 3D image size                             2048x2048x2048 pixels
    Max number of read image args                 128
    Max number of write image args                8
  Local memory type                               Local
  Local memory size                               32768 (32KiB)
  Local memory syze per CU (AMD)                  65536 (64KiB)
  Local memory banks (AMD)                        32
  Max number of constant args                     8
  Max constant buffer size                        4244635648 (3.953GiB)
  Preferred constant buffer size (AMD)            16384 (16KiB)
  Max size of kernel argument                     1024
  Queue properties                                 
    Out-of-order execution                        No
    Profiling                                     Yes
  Prefer user sync for interop                    Yes
  Profiling timer resolution                      1ns
  Profiling timer offset since Epoch (AMD)        1588525617364601339ns (Sun May  3 17:06:57 2020)
  Execution capabilities                           
    Run OpenCL kernels                            Yes
    Run native kernels                            No
   Thread trace supported (AMD)                  Yes
    Number of async queues (AMD)                  2
    Max real-time compute queues (AMD)            0
    Max real-time compute units (AMD)             96
    SPIR versions                                 1.2
  printf() buffer size                            4194304 (4MiB)
  Built-in kernels                                 
  Device Extensions                               cl_khr_fp64 cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_ato
mics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_
image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_vec3 cl_amd_printf cl_amd_med
ia_ops cl_amd_media_ops2 cl_amd_popcnt cl_khr_image2d_from_buffer cl_amd_bus_addressable_memory cl_khr_spir cl_khr_gl_event 

NULL platform behavior
  clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  No platform
  clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   No platform
  clCreateContext(NULL, ...) [default]            No platform
  clCreateContext(NULL, ...) [other]              Success [AMD]
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT)  Success (1)
    Platform Name                                 AMD Accelerated Parallel Processing
    Device Name                                   Ellesmere
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  Success (2)
    Platform Name                                 AMD Accelerated Parallel Processing
    Device Name                                   Ellesmere
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  Success (2)
    Platform Name                                 AMD Accelerated Parallel Processing
    Device Name                                   Ellesmere

After you verified that the driver works and the device is usable, the container needs to be restarted for foldingathome to recognize it. Exit the shell with ctrl+D (or type exit), stop the container, and start it again: docker stop foldingathome && docker start foldingathome

The GPU should now be available in foldingathome.

It is obviously nicer to put this into a Dockerbuild. You’re free to do that.

Making Linux responsive aka how IO priority and buffer size influences responsiveness

When doing backups on Linux, it is ususally wanted that the system stays responsive to inputs. That includes terminals and Xorg/Wayland. In the context of IO niceness, the concept of “everything is a file” bites Xorg/Wayland into the ass quite well because it turns out that writing to the frame buffers and refreshing the screen is also subject to IO classes and niceness. In conjunction with the fact that by default the Linux kernel has relatively large dirty page buffers and seems to work on the buffers in a single thread, this leads to Xorg and Wayland being blocked by tasks that have a large buffer to write back.

Continue reading “Making Linux responsive aka how IO priority and buffer size influences responsiveness”