Sway is an i3-compatible Wayland compositor. It is designed to be a drop-in replacement for i3 window manager, but works under Wayland environment.

Installation

Fedora now has a Sway Spin that comes with sway preinstalled and configured.
Alternatively, it could be installed on Fedora Workstation and other spins.

sudo dnf install "@Sway Desktop"
sudo dnf insatll "@Sway Window Manager (supplemental packages)" --allowerasing

Spin Bundled Components

Fedora Sway Spin comes with multiple components, including:

• swaylock: Simple screen locker.
• waybar: Highly customizable Wayland bar for Sway and Wlroots based compositors.
• rofi: Highly-configurable and lightweight notification daemon.
• dunst: Highly-configurable and lightweight notification daemon.
• kanshi: A Dynamic display configuration tool.

Sway Window Manager

The default Sway configuration is at /etc/sway/config. On Fedora Sway Spin, it reads additional configurations from 3 locations:

• /usr/share/sawy/config.d
• /etc/sway/config.d
• ~/.config/sway/config.d ($XDG_CONFIG_HOME/sway/config.d to be precise)

Kanshi

Fedora Sway Spin bundles kanshi, which is a dynamic display configuration tool.

Its configuration file is placed at ~/.config/kanshi/config. Multiple profiles could be defined in the configuration, one of which will be enabled if all of the listed outputs are connected.

Adding the first profile could be very straight forward. List all outputs recognized by sway:

swaymsg -t get_outputs

Take mine as an example, I have eDP-1 which is the laptop internal display, along with DP-1 and DP-2 which are  connected to my dock.
I am going disable the internal display and set the 2 external ones to side by side with scale:

profile {
    output eDP-1 enable scale 1.5
}

profile {
  output eDP-1 disable 
  output DP-1 enable mode 3840x2160@60 position 0,0 scale 2.0
  output DP-2 enable mode 3840x2160@60 position 1920,0 scale 2.0
}

Note that I am using 2X scaling (scale 2.0), so the position should be 3840 / 2 = 1920.
Finally let Kanshi start with Sway.
Add a Sway configuration segment to ~/.config/sway/config.d/10-kanshi.conf:

exec_always {
  pkill kanshi
  kanshi &
}

Tweaks and Customization

Override Default Terminal Emulator

I personally perfer Terminator over the default foot. So I added a new segmented configuration file at ~/.config/sway/config.d/10-preferred-applications.conf:

set $term terminator
bindsym --no-warn $mod+Return exec $term

Map Caps Lock to Control

I personally more adopted to Unix keyboard layout, especially no Caps Lock but a left control.
Add a new configuration to ~/.config/sway/config.d/10-input-keyboard.conf:

input "type:keyboard" {
	xkb_options caps:ctrl_modifier
}

Touchpad Tweaks

By default, there is no natural scrolling and tap to click. Add a new configuration to ~/.config/sway/config.d/10-input-touchpad.conf:

input "type:touchpad" {
       dwt enabled
       tap enabled
       natural_scroll enabled
       middle_emulation enabled
}

Software Integration

Most of the software work under Wayland or XWayland out of the box. But some of them still need fine-tuning.

Chromium

Chromium defaults to auto-select platform backend, which might lead to falling back to X. Visit chrome://flags and select Wayland for #ozone-platform-hint to indicate Chromium to use Wayland.

Electron in Flatpak

Some Flatpaks come with Wayland disabled by default because it might cause issues on non-HiDPI displays. When using a HiDPI monitor with scaling, however, this results in blurred user interface. To fix this, expose wayland socket to the application:

flatpak override --socket=wayland <APP_ID>

Alternatively, this could also be done in a GUI way via Featseal.

More to Read

• Fedora Sway Spin
• swaywm/sway Wiki
• Multihead - swaywm/sway Wiki
• Keyboard Layout - swaywm/sway Wiki
• Clamshell Mode - swaywm/sway Wiki
• kanshi
• flathub/com.slack.Slack  #212 Wayland issues on HiDPI

FreeDOS is free DOS - not only as in free beer, but also as in free will. FreeDOS provides Live CD and USB installer, but how if one wants Live USB? Since FreeDOS is simply a DOS distribution which is a simple DOS kernel with additional programs shipped, we can install the transfer the system and install packages manually.

Prepare the USB Drive

This could be done in a traditional (and DOS) way:

FDISK /AUTO 1
FORMAT /S
REM Assume the disk is C: drive
SYS C:

But a quick way to prepare the USB drive is to use Rufus: select the drive and choose "FreeDOS" in the image. Then you are a go!

Install the Base System

At this point, the USB drive is already bootable (of course, not UEFI but under legacy mode). We've already had a minimal system. Now, we are going to prepare a FreeDOS Live CD, which contains all the files we need.

Transfer System Files

Copy and rename CDROM:\FREEDOS\BIN\KERNL386.SYS to \KERNEL.SYS. This is the FreeDOS kernel.

The FreeDOS kernel will look for FDCONFIG.SYS to configure the system. Copy and rename CDROM:\FDOS_X86\FREEDOS\CONFIGS\CONFIG.DEF to \FDCONFIG.SYS. Open the file and replace the following variables:

• $FTARGET$ to C:\FREEDOS
• $FDRIVE$ to C:
• $FCCC$ to 001, $FCKC$ to 858 (or any other country code you would like to set)

And we also copy and rename CDROM:\FDOS_X86\FREEDOS\CONFIGS\AUTOEXEC.DEF to \FDAUTO.BAT. Then, again, replace:

• $FTARGET$ to C:\FREEDOS
• $FDRIVE$ to C:
• $FLANG$ to EN
• $TZ$ to HKT (or any time zone using abbreviation)
• $LANGSET$ to "" (delete the line)

Transfer Command Shell and Package Manager

Change directory to CDROM:\FDOS_X86\FREEDOS\BIN, copy command.com, fdimples.exe, fdinst.exe and fdnpkg.conf to \FREEDOS\BIN.

Now the base system is ready to use. There is a last step: copy PACKAGES folder from CD to \. Make sure you include all desired packages in the folder so that they could be installed later.

install Software Packages

Reboot the computer. Boot into FreeDOS and Choose 5 - Load FreeDOS without drivers (Emergency Mode). Start the package manager:

set TEMP=\TEMP
fdimples

To install FreeDOS, mark "Base" group. Optionally, install "V8Power" package from group "Utilities" for some text UIs.

Trouble Shooting

Unable to Use FDIMPLES

If the error occurs during installation process:

Error: custom dir 'links' is not a valid absolute path

Make sure you set an absolute (not relative) path in %DOSDIR%.

GPU passthrough on Proxmox VE host enable the guest to provide bare metal like experience - no RDP, no streaming. This creates a limit that per GPU could only be used in 1 guest, but it is still powerful while easier to configure and requires less advanced GPU.

Before we start, make sure host device passthrough and GPU passthrough is configured. If they are not, follow the first article in the series:

Proxmox VE for Workstation: Configure GPU Passthrough

Proxmox VE is an 0pen-source software server for virtualization management. It is chiefly designed for servers, but it is also great for workstations. Imagine you can have separated workspaces that run different operating systems, and they can run simultaneously!

# journalctl -xeu "Sakuragawa"Sakuragawa Asaba

Install Operating System and Drivers

Create a virtual machine for Windows:

export VMID=100
qm create $VMID --name windows-ltsc \
                --ostype win10 \
                --sockets 1 --cores 8 \
                --memory 24576 \
                --net0 virtio,bridge=vmbr0,firewall=1 \
                --scsi local-lvm:200 \
                --cdrom0 <Windows Installation ISO> \
                --cdrom1 <Virtio Drivers ISO> \
                --boot order=cdrom0\;scsi0

Then start the machine by qm start $VMID and finish the installation of Windows.

Now, at this point only VirtIO drivers need to be installed after Windows installation since no GPU passthrough is configured.

Setup GPU Passthrough and Install Drivers

Add GPU as a PCI device and disable built-in display:

export GPU_LOC="0000\:01\:00.0"
qm set $VMID --hostpci0 device-id=${GPU_LOC},pcie=1,x-vga=1
qm set $VMID --vga type=none

Connect an external monitor to the GPU and start the VM instance. Now Windows should pick up the GPU and display something on the external monitor, and we can proceed to GPU driver installation.

Issue Hunting

Fix mmap Failure on PVE 7.2+

On PVE 7.2 and later, mmap might fail. To fix the problem:

#!/bin/bash
echo 1 > /sys/bus/pci/devices/${GPU_LOC}/remove
echo 1 > /sys/bus/pci/rescan

Hide KVM from Guest

PVE already hides itself from NVIDIA driver out of the box. Yet we still need to set hv_vendor_id and kvm=off to run some specified games:

qm set $VMID -args "-cpu 'host,-hypervisor,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,kvm=off,hv_vendor_id=Proxmox'"

We can use qm showcmd $QMID to check the actual qemu command-line generated by PVE.

References

• Ultimate Beginner’s Guide to Proxmox GPU Passthrough
• Proxmox PCI(e) Passthrough in 2 minutes
• [SOLVED] AER PCIe Bus Errors
• [SOLVED] GPU Passthrough Issues After Upgrade to 7.2
• Hide vm from guest

Proxmox VE is an open-source software server for virtualization management. It is a hosted Type-2 hypervisor, and it can host Linux, as well as Windows and macOS. It is chiefly designed for servers, but it is also great for workstations. Imagine you can have separated workspaces that run different operating systems, and they can run simultaneously!

Proxmox VE is based on Debian GNU/Linux, so this part also works™️ for Debian GNU/Linux.

Prerequisite

• Enable VT-d in BIOS

Enable PCI(e) Passthrough

Edit /etc/default/grub to enable IOMMU:

....
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on video=efifb:off video=vesafb:off pcie_aspm=off"
....

Load VFIO kernel modules in /etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Finally, regenerate initramfs and reboot:

update-grub
update-initramfs -u -k all

Configure GPU Passthrough

Blacklist graphical card drivers to prevent them from being loaded:

echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

Check for the identifier of the GPU:

root@pve-workstation:~# lspci -v | grep VGA
01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller]

In this case, the identifier of GPU is 01:00.0. Then, check for the device ID:

root@pve-workstation:~# lspci -n -s 01:00
01:00.0 0300: 10de:1b81 (rev a1)
01:00.1 0403: 10de:10f0 (rev a1)

Here we have 2 device IDs (GPU and built-in audio). We need to configure VFIO for both of them:

export GPU_PCI="01:00"
export GPU_G_DID="$(lspci -n | grep '$GPU_PCI.0' | cut -d ':' -f 3-4 | cut -c 2-10)"
export GPU_A_DID="$(lspci -n | grep '$GPU_PCI.1' | cut -d ':' -f 3-4 | cut -c 2-10)"
echo "options vfio-pci ids=${GPU_G_DID},${GPU_A_DID} disable_vga=1"> /etc/modprobe.d/vfio.conf

To prevent the unsupported MSRS error which leads to crashes, add some tweaks to modules:

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf

Now, after a reboot, the graphics card should be ready to be passed through.

References

• Ultimate Beginner's Guide to Proxmox GPU Passthrough
• [SOLVED] AER PCIe Bus Errors
• [SOLVED] GPU Passthrough Issues After Upgrade to 7.2

Update to 22 failed with database error - Updated

That’s a pretty good idea anyways. I always test new versions of NC and dependencies before I roll them out in “production”. And that although I only use Nextcloud for personal things. But since I also migrated my girlfriend from Goggle to my NC, I can’t afford downtime anymore 😃

Nextcloud communityprivatewolke

Database Incompatibility with MariaDB 10.6.0 · Issue #1492 · nextcloud/docker

Hey! So I updated mariadb, which I use with nextcloud to version alpha 10.6.0. Which this verison this came https://mariadb.com/kb/en/innodb-system-variables/#innodb_read_only_compressed and gave m...

GitHubnextcloud

Temporarily disable read-only compressed row in MariaDB:

SET GLOBAL innodb_read_only_compressed=OFF;

Then patch the Nextcloud files manually and start upgrade from inside the container:

$ kubectl -n nextcloud exec statefulset/nextcloud -- /bin/bash
# rsync -rlDog --chown www-data:root --delete --exclude-from=/upgrade.exclude /usr/src/nextcloud/ /var/www/html/
# rsync -rlDog --chown www-data:root --include '/version.php' --exclude '/*' /usr/src/nextcloud/ /var/www/html/
# su - www-data -s /bin/bash -c 'php -d memory_limit=-1 /var/www/html/occ -vvv upgrade

To eventually fix this problem, we need to update row type from COMPRESSED to DYNAMIC on all tables. In MariaDB:

SELECT CONCAT('ALTER TABLE `', table_name, '` ROW_FORMAT=DYNAMIC;') AS aQuery FROM information_schema.tables WHERE table_schema = 'nextcloud'

Then execute the output as commands.

Nextcloud might add support for MariaDB 10.6 and fix this issue in later version of 23.

Add MariaDB 10.6 support (drop row_format=compressed) by acsfer · Pull Request #30129 · nextcloud/server

Keeping MariaDB 10.4 too as both versions have some BC breaks, so tests will run on both (for now).

GitHubnextcloud

The official FreeIPA container image requires a one-time installation process before running. For installation, a file containing ipa-server-install options should be provided, and Docker command should be ipa-server-install -U.

To complete this one-time process, create a docker-compose YAML file:

version: "3"
services:
  "central":
    container_name: "ipa-central"
    image: "docker.io/freeipa/freeipa-server:${FREEIPA_VERSION:-centos-8-stream}"
    command: "ipa-server-install -U"
    volumes:
      - "/sys/fs/cgroup:/sys/fs/cgroup:ro"
      - "./ipa-central/data:/data"
      - "./ipa-central/freeipa.options:/data/ipa-server-install-options"
    read_only: true
    hostname: "${FREEIPA_HOSTNAME}"
    sysctls:
      net.ipv6.conf.all.disable_ipv6: 0

Then start the process by docker-compose -f install.yml up. After installation is success, start the FreeIPA server container with docker-compose -f run.yml up:

version: "3"
services:
  "central-ipa":
    container_name: "ipa-central"
    image: "docker.io/freeipa/freeipa-server:${FREEIPA_VERSION:-centos-8-stream}"
    volumes:
      - "/sys/fs/cgroup:/sys/fs/cgroup:ro"
      - "./ipa-central/data:/data"
    read_only: true
    ports:
      - 127.0.0.1:9443:443
      - 389:389
      - 636:636
      - 88:88
      - 88:88/udp
      - 464:464
      - 464:464/udp
      - 123:123/udp
    hostname: "${FREEIPA_HOSTNAME}"
    sysctls:
      net.ipv6.conf.all.disable_ipv6: 0

Post-installation Setup

LDAP Service Account

There are some LDAP clients that need a pre-configured account. Do not use the Directory Manager account to authenticate remote services to the IPA LDAP server. A service account could be created like this:

[root@ipa /]# ldapmodify -x -D 'cn=Directory Manager' -W
Enter LDAP Password: <FreeIPA Directory Password>
dn: uid=authenticator,cn=serviceaccounts,cn=etc,dc=sakuragawa,dc=cloud
changetype: add
objectclass: account
objectclass: simplesecurityobject
uid: authenticator
userPassword: <password>
passwordExpirationTime: 20380119031407Z
nsIdleTimeout: 0

adding new entry "uid=authenticator,cn=serviceaccounts,cn=etc,dc=sakuragawa,dc=cloud"
<Ctrl-D>

The reason to use an account like this rather than creating a normal user account in IPA and using that is that the system account exists only for binding to LDAP. It is not a real POSIX user, can’t log into any systems and doesn’t own any files.

This use also has no special rights and is unable to write any data in the IPA LDAP server, only read.

References:

HowTo/LDAP - FreeIPA

FreeIPA

noahbliss/freeipa-sam

System Account Manager for FreeIPA. Contribute to noahbliss/freeipa-sam development by creating an account on GitHub.

GitHubnoahbliss

Allow Anonymous to Read mail Attribute

By default only bound (or authenticated) users are allowed to read mail attribute. For internal usage, querying the attribute anonymously is safe.

$ ipa permission-add 'Mail Readable by Anonymous' --type=user --attrs=mail --bindtype=anonymous --permissions=read

Upgrade Containerized FreeIPA

theoretically, after image is changed, FreeIPA container will start upgrade process automatically. But there was some common issues during the upgrade.

DIRSRV Failed to Start

2023056 – After upgrading ipa package, ipa server fails to start with dirsrv init error (/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so: undefined symbol: gost_yescrypt_pwd_storage_scheme_init)

So if an instance was created with early 8.5 builds, a plugin entry (dn: cn=GOST_YESCRYPT,cn=Password Storage Schemes,cn=plugins,cn=config) was created. Then the upgrade removed the init callback and startup fails.

A quick relief is by editing dse.ldif and removing cn=GOST_YESCRYPT,cn=Password Storage Schemes,cn=plugins,cn=config.

SSSD Failed to Start

988207 – sssd does not detail which line in configuration is invalid

SSSD will fail to start if the permission and owner of configuration  file at/etc/sssd/sssd/.conf isn't set properly.

# chown root /etc/sssd/sssd.conf
# chgrp root /etc/sssd/sssd.conf
# chmod 600  /etc/sssd/sssd.conf

Deploy KeyCloak Container

version: "3"
services:
  "central-sso":
    container_name: "keycloak-central"
    image: "quay.io/keycloak/keycloak:${KEYCLOAK_VERSION:-14.0.0}"
    ports:
      - 127.0.0.1:9080:8080
      - 127.0.0.1:9990:9990
    environment:
      DB_VENDOR: "mariadb"
      DB_ADDR: "db-master.example.com"
      DB_PORT: "3306"
      DB_DATABASE: "keycloak_skg"
      DB_USER: "keycloak"
      DB_PASSWORD: "${DB_PASSWORD}"
      PROXY_ADDRESS_FORWARDING: "true"
      KEYCLOAK_USER: "admin"
      KEYCLOAK_PASSWORD: "${KEYCLOAK_PASSWORD}"

Note that it is important to set PROXY_ADDRESS_FORWARDING=true in environment variables especially when KeyCloak is served behind a reverse proxy.

Getting Invalid parameter: redirect_uri when installed with keycloak-operator

Hi I have been trying to install Keycloak on a K8s environment but with limited success so far. I just followed the instructions of the keycloak-operator: make cluster/prepare kubectl apply -f deploy/operator.yaml kubectl apply -f deploy/examples/keycloak/keycloak.yaml This worked fine and I got…

Keycloakfabricepipart

nginx ingress: Invalid parameter: redirect_uri on admin access - helm-charts

helm-charts

Securing Services Behind OAuth2Proxy

Generate a cookie secret first:

$ export OAUTH2_PROXY_COOKIE_SECRET=`python -c 'import os,base64; print(base64.urlsafe_b64encode(os.urandom(16)).decode())'`

Then we will create a sakuragawa-sso.env to store all authentication backed related environment variables. Since there is still issues with Keycloak backend, we will turn to use OpenID Connect (OIDC) backend here.

Issues with Keycloak integration · Issue #1093 · oauth2-proxy/oauth2-proxy

Integrating oauth2-proxy with Keycloak is not worknig. When I inspected the logs, it says Invalid parameter value for: scope Expected Behavior Must be able to login to the respective service using ...

GitHuboauth2-proxy

Keycloak Provider is just Barebones OAuth - Doesn’t Expose OIDC Potential · Issue #956 · oauth2-proxy/oauth2-proxy

We have a keycloak provider that most Keycloak users likely gravitate to by default. It is a very barebones implementation mostly riding our default OAuth provider methods, only overriding the User...

GitHuboauth2-proxy

OAUTH2_PROXY_PROVIDER=oidc
OAUTH2_PROXY_PROVIDER_DISPLAY_NAME=Sakuragawa SSO
OAUTH2_PROXY_CLIENT_ID=<client ID>
OAUTH2_PROXY_CLIENT_SECRET=<client secret>
OAUTH2_PROXY_OIDC_ISSUER_URL=https://sso.central.sakuragawa.cloud/auth/realms/SKG
OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL=true
OAUTH2_PROXY_ALLOWED_GROUPS=<keycloak group 1>, <keycloak group 2>

Then create an orchestration file which includes global settings:

version: "3"
services:
  auth:
    image: "quay.io/oauth2-proxy/oauth2-proxy:latest"
    ports:
      - "127.0.0.1:8180:4180"
    environment:
      OAUTH2_PROXY_UPSTREAMS: "http://<upstream>:5000/"
      OAUTH2_PROXY_WHITELIST_DOMAINS: ".central.sakuragawa.cloud:*"
      OAUTH2_PROXY_HTTP_ADDRESS: ":4180"
      OAUTH2_PROXY_COOKIE_SECRET:
      OAUTH2_PROXY_REVERSE_PROXY: "true"
      OAUTH2_PROXY_EMAIL_DOMAINS: "example.com,example.org"
      OAUTH2_PROXY_PASS_USER_HEADERS: "true"
    env_file:
      - sakuragawa-sso.env
    restart: "unless-stopped"

Ability to ignore unverified email · Issue #117 · oauth2-proxy/oauth2-proxy

Keycloak doesn't verify email for users imported from LDAP https://stackoverflow.com/questions/38307029/ldap-federated-users-automatic-set-email-verified-to-true So, I have to change default ma...

GitHuboauth2-proxy

References

Federation with Keycloak and FreeIPA - Red Hat Customer Portal

Red Hat Customer Portal

Gitea SSO with Keycloak, OpenLDAP and OpenID Connect

Blog by Ben Dixon, Ruby on Rails Developer, about rails, kubernetes, docker, climbing and startups

Catapult

Configure SAML Authentication for Nextcloud with Keycloack

Introduction The complex problems of identity and access management (IAM) have challenged big companies and in result we got powerful protocols, technologies and concepts such as SAML, oAuth, Keycloack, tokens and much more. The goal of IAM is simple. Centralize all identities, policies and get ri…

Configure SAML Authentication for Nextcloud with Keycloack

Unable to retsart after disk-full · Issue #3591 · etcd-io/etcd

Hi, I'm unable to restart a standalone etcd2 server after a disk full event on CoreOS stable 723.3.0. I upgraded CoreOS to the latest stable (766.3.0) as it has etcd 2.1 which should recovers f...

GitHubetcd-io

Round 1: etcdmain: read .wal: input/output error

2021-11-21 20:43:04.262587 I | embed: initial cluster =
2021-11-21 20:43:04.280202 W | wal: ignored file 000000000000035c-00000000045d83a6.wal.broken in wal
2021-11-21 20:43:04.280256 W | wal: ignored file 0000000000000375-0000000004764aca.wal.broken in wal
2021-11-21 20:43:08.251726 C | etcdmain: read /var/lib/rancher/etcd/member/wal/0000000000000375-0000000004764aca.wal: input/output error

We can find the missing file is renamed from .wal to .wal.broken. We could rename the file back to .wal.

Round 2: etcdmain: walpb: crc mismatch

2021-11-21 20:50:19.312596 I | embed: initial cluster =
2021-11-21 20:50:19.323572 W | wal: ignored file 000000000000035c-00000000045d83a6.wal.broken in wal
2021-11-21 20:50:19.507366 C | etcdmain: walpb: crc mismatch

To solve this problem, one choice is to save and restore a snapshot.

Before starting, we could get some preparation:

# export NODE_NAME=container-1-paas-central-sakuragawa-cloud
# export KUBE_CA=/etc/kubernetes/ssl/kube-ca.pem
# export ETCD_PEM=/etc/kubernetes/ssl/kube-etcd-$NODE_NAME.pem
# export ETCD_KEY=/etc/kubernetes/ssl/kube-etcd-$NODE_NAME-key.pem
# export ETCD_SERVER=https://container-2:2379
# alias etcdctl="etcdctl --cert=$ETCD_PEM --key=$ETCD_KEY --cacert=$KUBE_CA --endpoints=$ETCD_SERVER"

Save a snapshot from other working nodes:

# etcdctl snapshot save snapshot.db
{"level":"info","ts":1637576476.7098856,"caller":"snapshot/v3_snapshot.go:68","msg":"created temporary db file","path":"snapshot.db.part"}
{"level":"info","ts":1637576476.722268,"logger":"client","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1637576476.7223496,"caller":"snapshot/v3_snapshot.go:76","msg":"fetching snapshot","endpoint":"https://container-2:2379"}
{"level":"info","ts":1637576478.531543,"logger":"client","caller":"v3/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":1637576479.3304148,"caller":"snapshot/v3_snapshot.go:91","msg":"fetched snapshot","endpoint":"https://container-2:2379","size":"25 MB","took":"2 seconds ago"}
{"level":"info","ts":1637576479.3305554,"caller":"snapshot/v3_snapshot.go:100","msg":"saved","path":"snapshot.db"}
Snapshot saved at snapshot.db

Then stop etcd server and restore the snapshot:

# docker stop etcd
# mv /var/lib/etcd/ /var/lib/etcd.old/
# etcdctl snapshot restore snapshot.db --data-dir=/var/lib/etcd/

Round 3: Restore from Snapshot (Again)

After etcd starts, it is not back to work as expected:

2021-11-22 11:46:37.003240 E | rafthttp: request cluster ID mismatch (got e8061948cdccadb7 want cdf818194e3a8c32)

First stop the node agent (again) and etcd container and delete all the etcd local data:

# docker stop etcd
# rm -f /var/lib/etcd/member/wal/*
# rm -f /var/lib/etcd/member/snap/*

To restore the node, we need some parameters from the origin node. We can run docker inspect etcd and collect some information from Args and Envs :

• --name
• --initial-cluster
• --initial-cluster-token
• --initial-advertise-peer-urls

Then use these parameters when restoring from a snapshot:

# etcdctl snapshot restore snapshot.db \
	--cert=$ETCD_PEM \
	--key=$ETCD_KEY \
    --cacert=$KUBE_CA \
    --name=etcd-container-1 \
    --initial-cluster=etcd-container-1="https://container-1:2380,etcd-container-2=https://container-2:2380" \
    --initial-cluster-token="etcd-cluster-1" \
    --initial-advertise-peer-urls="https://container-1:2380" \
    --data-dir /var/lib/etcd
Deprecated: Use `etcdutl snapshot restore` instead.

2021-11-22T21:07:11+08:00       info    netutil/netutil.go:112  resolved URL Host       {"url": "https://container-1:2380", "host": "container-1:2380", "resolved-addr": "192.168.1.21:2380"}
2021-11-22T21:07:11+08:00       info    netutil/netutil.go:112  resolved URL Host       {"url": "https://container-1:2380", "host": "container-1:2380", "resolved-addr": "192.168.1.21:2380"}
2021-11-22T21:07:11+08:00       info    snapshot/v3_snapshot.go:251     restoring snapshot      {"path": "snapshot-20211122.db", "wal-dir": "/var/lib/etcd/member/wal", "data-dir": "/var/lib/etcd", "snap-dir": "/var/lib/etcd/member/snap", "stack": "go.etcd.io/etcd/etcdutl/v3/snapshot.(*v3Manager).Restore\n\t/tmp/etcd-release-3.5.1/etcd/release/etcd/etcdutl/snapshot/v3_snapshot.go:257\ngo.etcd.io/etcd/etcdutl/v3/etcdutl.SnapshotRestoreCommandFunc\n\t/tmp/etcd-release-3.5.1/etcd/release/etcd/etcdutl/etcdutl/snapshot_command.go:147\ngo.etcd.io/etcd/etcdctl/v3/ctlv3/command.snapshotRestoreCommandFunc\n\t/tmp/etcd-release$3.5.1/etcd/release/etcd/etcdctl/ctlv3/command/snapshot_command.go:128\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/remote/sbatsche/.gvm/pkgsets/go1.$6.3/global/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/remote/sbatsche/.gvm/pkgsets/go1.16.3/$lobal/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global$pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897\ngo.etcd.io/etcd/etcdctl/v3/ctlv3.Start\n\t/tmp/etcd-release-3.5.1/etcd/release/etcd/etcdctl/ctlv3/ctl$go:107\ngo.etcd.io/etcd/etcdctl/v3/ctlv3.MustStart\n\t/tmp/etcd-release-3.5.1/etcd/release/etcd/etcdctl/ctlv3/ctl.go:111\nmain.main\n\t/tmp/etcd-release-3.$.1/etcd/release/etcd/etcdctl/main.go:59\nruntime.main\n\t/home/remote/sbatsche/.gvm/gos/go1.16.3/src/runtime/proc.go:225"}
2021-11-22T21:07:12+08:00       info    membership/store.go:141 Trimming membership information from the backend...
2021-11-22T21:07:12+08:00       info    membership/cluster.go:421       added member    {"cluster-id": "e8061948cdccadb7", "local-member-id": "0", "added-p$er-id": "94447210df55f5df", "added-peer-peer-urls": ["https://container-2:2380"]}
2021-11-22T21:07:12+08:00       info    membership/cluster.go:421       added member    {"cluster-id": "e8061948cdccadb7", "local-member-id": "0", "added-p$er-id": "e089ac38e0c2282f", "added-peer-peer-urls": ["https://container-1:2380"]}
2021-11-22T21:07:12+08:00       info    snapshot/v3_snapshot.go:272     restored snapshot       {"path": "snapshot-20211122.db", "wal-dir": "/var/lib/etcd/$ember/wal", "data-dir": "/var/lib/etcd", "snap-dir": "/var/lib/etcd/member/snap"}

Finally, start etcd and the node will be back to the cluster.

$ docker start etcd
$ docker exec -e ETCDCTL_ENDPOINTS=$(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','") etcd etcdctl endpoint status --write-out table
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://container-2:2379 | 94447210df55f5df |  3.4.14 |   25 MB |      true |      false |     82444 |   75080617 |           75080617 |        |
| https://container-1:2379 | e089ac38e0c2282f |  3.4.14 |   25 MB |     false |      false |     82444 |   75080617 |           75080617 |        |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

References

Troubleshooting etcd Nodes

Rancher Labs

I decided to build a NAS with old hard drives chiefly for serving media files like movie, TV shows and music.

When building a multi-bay system, it's inevitable to consider which kind of disk redundancy to use. Traditional RAID (both hard and soft ones) would work, but it also have several cons:

• Data are distributed onto disks so that it could be not read from a single drive
• Requires several disks to work simultaneously when reading or writing
• If all parity disks plus 1 more disk corrupt, all data will be lost

For home data storage, I'd prefer scheduled cold backup. But is there any way to enjoy some benefits from RAID without "real RAID"? Yes, there is!

Backgrounds

MergerFS

MergerFS could pool multiple filesystems or folders into one place. It allows us to create RAID0-like filesystems but keeps data intact when some drives are broken.

mergerfs is a union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices. It is similar to mhddfs, unionfs, and aufs.

SnapRAID

SnapRAID is short for "Snapshot RAID". It is a backup program for disk arrays and capable of recovering from up to six disk failures. It is mainly targeted for a home media center, with a lot of big files that rarely change.

Some highlights of SnapRAID are:

• The disks can have different sizes
• Add disks at any time
• Quit SnapRAID without moving data and formatting disks
• Anti-delete function

Differences between mergerfsfolders & unionfilesystems

There are 2 similar plugins based on mergerfs in OMV-Extras:

• mergerfsfolders pools folders
• unionfilesystems pools drives

With  blank new drives here, we would use unionfilesystems for simplified configuration.

Prepare System and Drives

I have 4×2TiB hard drives and plan to use 3 of them for data and 1 for parity check.

In OpenMediaVault:

1. Install and configure OpenMediaVault as needed.
2. Install OMV-Extras repository.
3. Install openmediavault-snapraid and openmediavault-unionfilesystem plugin.
4. Wipe all the disks and create filesystems on them.

I also labeled all file systems with DXX for data disks and PXX for the lonely parity disk.

Configure MergerFS

In OpenMediaVault control panel:

1. Navigate to Storage > Union Filesystems.
2. Click on Add.
3. Type a pool name in Name.
4. Check all data disks (but exclude parity disks) in Branches.
5. Select Existing path, most free space (or epmfs) in Create Policy.
6. Save and apply.

Here I set create policy to epmfs because I want the files gather together as possible as they could be. Under epmfs policy, new files will be created together in the same drive when the upper level folder path exists. You can check other available create policies upon your needs.

Note that some applications (`libttorrent` applications, sqlite3, etc.) requires mmap. Since I would use qBittorrent, I need to add some options:

allow_other,use_ino,cache.files=partial,dropcacheonclose=true,...

Also as I have labeled data disks as D01, D02 and D03, I expect MergerFS write files into them in that given order. Unfortunately this order could not be specified or modified via the control panel. I have to edit /etc/fstab:

....
/srv/dev-disk-by-uuid-${D01}:/srv/dev-disk-by-uuid-${D02}:/srv/dev-disk-by-uuid-${D03}            /srv/${MERGERFS_UUID}       fuse.mergerfs   defaults,allow_other,cache.files=off,use_ino,category.create=ff,minfreespace=4G,fsname=${MERGERFS_NAME}:${MERGERFS_UUID},x-systemd.requires=/srv/dev-disk-by-uuid-${D01},x-systemd.requires=/srv/dev-disk-by-uuid-${D02},x-systemd.requires=/srv/dev-disk-by-uuid-${D03} 0 0
....

Save the file, umount and mount MergerFS mountpoint. Now the writing order is corrected.

Configure SnapRAID

First configure data disks. In OpenMediaVault control panel:

1. Navigate to Services > SnapRAID.
2. Navigate to Disks tab.
3. Click on Add.
4. Select a data disk for Disk.
5. Check Content and Data, then click on Save.

Repeat this until all data disks are added. Then configure parity disks:

1. Repeat step 1-4 when configuring data disks.
2. Check Parity , then click on Save.

Also repeat this until all parity disks are configured.

Initialize and Use SnapRAID

First Start

After SnapRAID is set up for the first time, you should synchronize the data, create and check parity. There are several useful commands:

• sync hashes files and save the parity.
• scrub verifies the data with the generated hashes and detect errors
• status shows the current array status

So in combined:

snapraid {sync, scrub, status}

Rules

Not every piece of data worth backing up. Most of the time for home usage, only "static files" need to be backed up. For example, there is no strong necessity to back up container images, caches and other temporary files. These directories or files could be excluded in Rules tab.

Scheduled Diff and Cron Jobs

Click on Scheduled diff button and SnapRAID plugin will generate a cron job in Scheduled Jobs tab. This job only contains diff operation which calculates file differences from the last snap. To maintain the parity synchronized,  sync and scrub jobs are also needed. If you plan to synchronize automatically, you would like to check a helper script when creating jobs.

Sometimes empty files could be created by programs and cause errors:

The file '${FILENAME}' has unexpected zero size!
It's possible that after a kernel crash this file was lost,
and you can use 'snapraid fix -f ${FILENAME}' to recover it.
If this an expected condition you can 'sync' anyway using 'snapraid --force-zero sync'

If that file seems to be fine with a zero content with you, simply add the --force-zero parameter to synchronize.

I have a ZFS pool containing 8 disks (4×2TiB + 2×4TiB + 2×4TiB) on an 8 bay server:

 state: DEGRADED
status: ....
config:

        NAME           STATE     READ WRITE CKSUM
        ${SRC_POOL}    DEGRADED     0     0     0
          raidz2-0     DEGRADED     0     0     0
            da0p2      ONLINE       0     0     0
            da1p2      DEGRADED     0     0     0
            da2p2      ONLINE       0     0     0
            da3p2      ONLINE       0     0     0
          mirror-1     ONLINE       0     0     0
            da4p2      ONLINE       0     0     0
            da5p2      ONLINE       0     0     0
          mirror-2     ONLINE       0     0     0
            da6p2      ONLINE       0     0     0
            da7p2      ONLINE       0     0     0

errors: No known data errors

When it comes to the time to replace the warning drive and expand the storage pool, I want to give up the current striped pool and create a new pool containing a single RAID-Z2 vdev (4×8TiB). Now here is the problem:

How to migrate create a new pool with all 8 bays are occupied?

The traditional way is to backup the content to external storage and restore. But here is a possible solution without any external storage: creating a degraded pool with minimum devices.

Tricky Drive Replacement

At least 2 online devices are required to keep a RAID-Z2 vdev working. So the trick is to pull out 2 drives from the original pool and insert 2 new blank drives.

+------+------+------+------+     +------+------+------+------+
|        RAID-Z2 4×2T       |     | NEW1 | NEW2 |RAID-Z2 2×2T |
|------+------+------+------| ==> |------+------+------+------|
| MIRROR 2×4T | MIRROR 2×4T |     | MIRROR 2×4T | MIRROR 2×4T |
+------+------+------+------+     +------+------+------+------+

Now the original pool enters DEGRADED status (it has been in degraded status anyway).

Create a Degraded Pool

Create some sparse files as "fake devices". The size of the sparse files should match the size of the hard drives (for any convenience).

SPARSE_SIZE=8T
SPARSE_FP_1=/root/sparsefile1
SPARSE_FP_2=/root/sparsefile2
truncate -s ${SPARSE_SIZE} ${SPARSE_FP_1}
truncate -s ${SPARSE_SIZE} ${SPARSE_FP_2}

# ls -l /root/sparsefile*
-rw-r--r--  1 root  wheel   8.0T Oct 26 04:15 sparsefile1
-rw-r--r--  1 root  wheel   8.0T Oct 26 04:15 sparsefile2

Create a ZFS Storage Pool at RAID Z2 with 2 hard drives and 2 sparse files:

zpool create ${POOL_NAME} raidz2 \
	da0p2 da1p2 \
    ${SPARSE_FP_1} \
    ${SPARSE_FP_2}
zfs set aclmode=passthrough ${POOL_NAME}
zfs set compression=lz4 ${POOL_NAME}

Set the sparse files to offline so that no actual data will be written:

zpool offline ${POOL_NAME} ${SPARSE_FP_1}
zpool offline ${POOL_NAME} ${SPARSE_FP_2}

root@storage-01:~ # zpool status SKG_STORAGE_1
  pool: ${POOL_NAME}
 state: DEGRADED
status: ....
config:

        NAME                                            STATE     READ WRITE CKSUM
        ${POOL_NAME}           DEGRADED     0     0     0
          raidz2-0             DEGRADED     0     0     0
            da0p2              ONLINE       0     0     0
            da1p2              ONLINE       0     0     0
            /root/sparsefile1  OFFLINE      0     0     0
            /root/sparsefile2  OFFLINE      0     0     0

errors: No known data errors

Now we have a degraded pool with 2 actual drives and 2 fake drives. Finally adjust export the pool so that it could be imported in FreeNAS Web UI:

zpool export ${POOL_NAME}

Take and Send Snapshots

Create a snapshot:

zfs snapshot -r ${SRC_POOL}@migration-base

Dry-run before sending the actual data and proceed if everything is checked:

zfs send -Rvn -i ${SRC_POOL}@migration_base
zfs send -Rv -i snapshot ${SRC_POOL}@migration-base | pv | zfs receive -Fsvd ${POOL_NAME}@migration-base

Replace Fake Devices with Real Ones

zpool replace ${POOL_NAME} ${SPARSE_FP_1} da3p2
zpool replace ${POOL_NAME} ${SPARSE_FP_2} da4p2

References

• How do I make a degraded 2 drive RAIDZ?
• how to use zfs send | zfs receive to back up my current pools to a 10TB transfer disk?

CloudFlare provides free CDN with full SSL support. It is powerful and userful. But sometimes one can run into problems with it.

TOO_MANY_REDIRECTS

When CloudFlare CDN is set to on, CloudFlare will be proxying access to the origin server. The connection between CloudFlare and origin server could be either HTTP or HTTPS. This could be controlled in SSL/TLS tab.

If SSL is set to Flexible, CloudFlare will connect to the origin server via HTTP. Now if origin server is configured to redirect all HTTP access to  HTTPS, CloudFlare will insist connecting via HTTP, and after multiple retries this error happens. Under this case, SSL should be set to Full (strict) which prompts CloudFlare to use HTTPS for connection to origin server.

ERR_SSL_VERSION_OR_CIPHER_MISMATCH

This is caused by Multi-Level Subdomain DNS. CloudFlare free tier provides SSL certificates for example.org and *.example.com. The 4th subdomains will not match any valid certificate due to a limitation in SSL.

To solve this problem, CloudFlare provides (paid) dedicated certificates. But if you stick with the free plan, you can only use 3rd subdomains with CloudFlare CDN.

It becomes so easy to host a WordPress CMS using its official Docker container image. If only 1 domain is available, it might be useful to serve WordPress from subdirectory, i.e https://$SITE_URL/blog/.

First, follow the official documentation (Method II) to setup WordPress installation in subdirectory.

Giving WordPress Its Own Directory

Many people want WordPress to power their website’s root (e.g. but they don’t want all of the WordPress files cluttering up their root directory. WordPress allows you to install it into…

WordPress.org Forums

$ export SUBPATH="blog"
$ mv !($SUBPATH) $SUBPATH/
$ cp SUBPATH/index.php index.php
$ sed -i "s/wp-blog-header.php/$SUBPATH\/wp-blog-header.php/g"

Then configure Traefik label:

version: '3'
services:
	wordpress:
    	....
        labels:
            - 'traefik.enable=true'
            - 'traefik.http.routers.blog.rule=Host(`$SITE_URL`) && PathPrefix(`/blog`)'
            - 'traefik.http.routers.blog.entrypoints=https'
        ....
        
            - 'traefik.http.routers.blog.tls=true'

References

Subdirectory problem (wp-admin redirect) · Issue #388 · docker-library/wordpress

I have looked through several related issues, but could not find a solution. It only happens when I'm at /wp-admin. I get a sort of 'redirect' when I access wp-admin. Seems like a pushS...

GitHubdocker-library

[URGENT NEED] [Please Help Needed A Lot] - Unable to route nginx to wordpress (fpm or apache) for /subdirectory config · Issue #221 · docker-library/wordpress

I have been trying this more than 12 hours and still not able to figure the perfect nginx config to route to a /blog wordpress installation. Need this in real urgent basis, have to shift company&#3...

GitHubdocker-library

Converting and changing the type of a column in database is easy. Unfortunately, the type of a field in a mapping could not be changed in Elastic. If the docs is already indexed, the mapping could not be changed and an reindex-ing would be needed.

Dynamic Mapping Failed Grafana

Elastic could be used as a data source for Grafana. But if there exists no field of date type in a index for Time field name, the data source could not be configured properly.

In my case, unfortunately, the type of timestamp field is set to text during dynamic mapping process due to inrecognizable time format (requires yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ but gets no "T" as a delimiter).

Explained in official documentation:

Except for supported mapping parameters, you can’t change the mapping or field type of an existing field. Changing an existing field could invalidate data that’s already indexed.

If you need to change the mapping of a field in other indices, create a new index with the correct mapping and reindex your data into that index.

Elasticsearch does not provide the functionality to change types for existing fields. A workaround is to import the data to a new index with correct types:

1. create a new index
2. put updated mapping (create manually a mapping where fields have the types you need before sending the first data) to the new index
3. use reindex API to copy data from the old index to the newly created one
4. (optionally) create an alias with the old index name pointing to the new index

Construct Explicit Mapping Request

First we should know the old index mapping and find out what is wrong:

$ curl "$ES_URL/$INDEX/_mapping/field/timestamp?pretty"

{
  "testrun": {
    "mappings": {
      "timestamp": {
        "full_name": "timestamp",
        "mapping": {
          "timestamp": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }
  }
}

We can clearly see a timestamp field with text type. This is what to expect, as Grafana requires date type. So we can construct a payload.json with explicit mapping:

{
  "properties": {
    "timestamp": {
      "type": "date",
      "format": "yyyy-MM-dd HH:mm:ss.SSSSSSZZZZZ"
    }
  }
}

Here we have a custom date format because our raw data is in non-standard format like 2017-11-30 11:59:19.717648+00:00.

Then we will create a new index and put an explicit mapping into it.

$ curl -X PUT $ES_URL/$INDEX_NEW
$ curl -X PUT $ES_URL/$INDEX_NEW/_mapping -H "Content-Type: application/json" -d @payload.json

Reindex from Old Index to New

Reindex API

Again we can create a payload.json:

{
  "source": {
    "index": "$INDEX"
  },
  "dest": {
    "index": "$INDEX_NEW"
  }
}

$ curl -X POST "$ES_URL/_reindex?pretty" -H 'Content-Type: application/json' -d @payload.json

If no errors are returned, reindex process is started. The process will take from minutes to hours, depending on the quantity of data. But by now Grafana should recognize the field and shows

Index OK. Time field name OK.

And the data source is sucessfully configured.

Reindex Asynchronously

Reindexing a huge index could take hours to days and will result in an HTTP request timeout. To make sure

If the request contains wait_for_completion=false, Elasticsearch performs some preflight checks, launches the request, and returns a task you can use to cancel or get the status of the task. Elasticsearch creates a record of this task as a document at _tasks/<task_id>

$ curl -X POST "$ES_URL/_reindex?wait_for_completion=false" -H 'Content-Type: application/json' -d @payload.json

This will returns a tasks represented by its ID:

{ "task": "Sb_1wXmiTWWY2uFEczPhIQ:868863502" }

And this task ID could be used to query the process:

$ export TASK_ID="Sb_1wXmiTWWY2uFEczPhIQ:868863502"
$ curl "$ES_URL/_tasks/$TASK_ID?pretty"

Any warning or error will be included in the query result. When you confirm the process is done and task logs are no longer needed, this task should be deleted. The tasks are store in an intenral index .tasks, and it is where the task document should be deleted from.

$ curl -X DELETE "$ES_URL/.tasks/task/$TASK_ID"

How About the Old Index?

There is no index renaming operation in Elastic. But by the experience we've gained, it is more eaiers to rename one by deleting, creating and reindexing.

There is also an option to create alias for the index.

Multi-Garment Net: Learning to Dress 3D People from Images

Multi-Garment Net: Learning to Dress 3D People from Images

We present Multi-Garment Network (MGN), a method to predict body shape andclothing, layered on top of the SMPL model from a few frames (1-8) of a video.Several experiments demonstrate that this representation allows higher level ofcontrol when compared to single mesh or voxel representations of s…

arXiv.orgBharat Lal Bhatnagar

bharat-b7/MultiGarmentNetwork

Repo for “Multi-Garment Net: Learning to Dress 3D People from Images, ICCV′19” - bharat-b7/MultiGarmentNetwork

GitHubbharat-b7

Basic environment: CentOS 7, NVIDIA drivers (CUDA 10.1.243), Conda 4.10.

I hate Conda!

Python 2 Environment

First attempt is with Python 2.7 since the readme says

The code has been tested in python 2.7, Tensorflow 1.13

Create Conda environment:

$ conda create -n mgn-py27 python=2.7 cudatoolkit=10.1 cudnn=7.6.5
$ conda activate mgn-py27

MultiGarment-Network has 2 dependencies that we need to install manually.

Install dirt

pmh47/dirt

DIRT: a fast differentiable renderer for TensorFlow - pmh47/dirt

GitHubpmh47

Install dependencies:

(mgn-py27)$ conda install cmake gcc_linux-64
(mgn-py27)$ conda install tensorflow-gpu=1.13

But pip list is not showing tensorflow-gpu package. So we need to install it again from pypi before building and installation:

(mgn-py27)$ pip install --ignore-installed tensorflow==1.13.1
(mgn-py27)$ pip install .

Install Mesh

MPI-IS/mesh

MPI-IS Mesh Processing Library. Contribute to MPI-IS/mesh development by creating an account on GitHub.

GitHubMPI-IS

(mgn-py27)$ conda install gxx_linux-64 opencv
(mgn-py27)$ git checkout 1761d544686b3735991954947a8befa759891eb4
(mgn-py27)$ make
(mgn-py27)$ cd dist && pip install psbody_mesh-0.1-cp27-cp27mu-linux_x86_64.whl

Run MultiGarment-Network

(mgn-py27)$ conda install matlabplot
(mgn-py27)$ pip install "scikit-learn<0.18" chumpy
(mgn-py27)$ python test_network.py
Using dirt renderer.
....
Done
freeglut (mesh_viewer):  ERROR:  Internal error <FBConfig with necessary capabilities not found> in function fgOpenWindow

Python 3 Environment (WIP)

Create a Conda environment:

$ conda create -n mgn-py36 python=3.6 cudatoolkit=10.0 cudnn=7.6.5
$ conda activate mgn-py36

Install dirt

Note the troubleshooting section in the description:

If you are using TensorFlow 1.14, there are some binary compatibility issues when using older versions of python (e.g. 2.7 and 3.5), due to compiler version mismatches. These result in a segfault at tensorflow::shape_inference::InferenceContext::GetAttr or similar. To resolve, either upgrade python to 3.7, or downgrade TensorFlow to 1.13, or build DIRT with gcc 4.8

Thus we will use an older version 1.13 of Tensorflow to avoid this known issue. And since the package from conda-forge messes up the dependencies, we must specify channel to anaconda:

(mgn-py36)$ conda install -c anaconda tensorflow-gpu=1.13
(mgn-py36)$ pip install tensorflow-gpu==1.13.1

Then install rest dependencies for building:

(mgn-py36)$ conda install cmake gcc_linux-64

Finally build and install dirt, but with some environment variables:

(mgn-py36)$ export CUDA_HOME=/usr/local/cuda-10.1
(mgn-py36)$ export PATH=$CUDA_HOME/bin:$PATH
(mgn-py36)$ pip install .
(mgn-py36)$ python tests/square_test.py
....
successful: all pixels agree

Install Mesh

(mgn-py36)$ conda install boost
(mgn-py36)$ conda install pyopengl pillow pyzmq pyyaml
(mgn-py36)$ conda install gxx_linux-64
(mgn-py36)$ make all
(mgn-py36)$ make tests
....
Ran 28 tests in 5.882s

Convert MultiGarment-Network to Python 3

(mgn-py36)$ conda install matplotlib scikit-learn==0.17 chumpy

Since we are running Python 3, we will need to replace cPickle with _pickle as the former does not exist.

(mgn-py36)$ find ./ -type f -name \*.py -exec sed -i -e 's/cPickle/_pickle/g' {} \;
(mgn-py36)$ 2to3 -w -n MultiGarmentNetwork/

Due to some incompatibility, Pickle un-serialization will fail:

Traceback (most recent call last):
  File "test_network.py", line 171, in <module>
    _, faces = pkl.load(f)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x8c in position 16: ordinal not in range(128)

You will need to change the Pickle loading to

_, faces = pkl.load(open(file_path, 'rb'), encoding='latin1')

A too-old scikit-learn is also a problem:

Traceback (most recent call last):
  File "test_network.py", line 176, in <module>
    pca_verts[garment] = pkl.load(f)
  File "$CONDA_HOME/lib/python3.6/site-packages/sklearn/decomposition/__init__.py", line 10, in <module>             
    from .kernel_pca import KernelPCA
  File "$CONDA_HOME/lib/python3.6/site-packages/sklearn/decomposition/kernel_pca.py", line 13, in <module>           
    from ..metrics.pairwise import pairwise_kernels
  File "$CONDA_HOME/lib/python3.6/site-packages/sklearn/metrics/__init__.py", line 33, in <module>                   
    from . import cluster
  File "$CONDA_HOME/lib/python3.6/site-packages/sklearn/metrics/cluster/__init__.py", line 8, in <module>
    from .supervised import adjusted_mutual_info_score
  File "$CONDA_HOME/lib/python3.6/site-packages/sklearn/metrics/cluster/supervised.py", line 14, in <module>         
    from scipy.misc import comb
ImportError: cannot import name 'comb'

Changing to from scipy.special import comb will solve the problem.

Until now the network should run without syntax error. Some report that the network could run after the above changes.

But here we will still get this error:

Traceback (most recent call last):
  File "test_network.py", line 185, in <module>
    pred = get_results(m, dat)
  File "test_network.py", line 55, in get_results
    out = m([images, vertex_label, J_2d])
  File "$CONDA_PREFIX/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 592, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/public/wl4/clothing/MultiGarmentNetwork-py3/network/base_network.py", line 336, in call
    garm_model_outputs = [fe(latent_code_offset_ShapeMerged) for fe in self.garmentModels]
  File "/public/wl4/clothing/MultiGarmentNetwork-py3/network/base_network.py", line 336, in <listcomp>
    garm_model_outputs = [fe(latent_code_offset_ShapeMerged) for fe in self.garmentModels]
  File "$CONDA_PREFIX/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 592, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/public/wl4/clothing/MultiGarmentNetwork-py3/network/base_network.py", line 65, in call
    x = self.PCA_(pca_comp)
  File "$CONDA_PREFIX/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 592, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/public/wl4/clothing/MultiGarmentNetwork-py3/network/custom_layers.py", line 33, in call
    return tf.reshape(tf.matmul(x, self.components) + self.mean, (-1, K.int_shape(self.mean)[0] / 3, 3))
  File "$CONDA_PREFIX/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 7161, in reshape
    tensor, shape, name=name, ctx=_ctx)
  File "$CONDA_PREFIX/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 7206, in reshape_eager_fallback
    ctx=_ctx, name=name)
  File "$CONDA_PREFIX/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 66, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Value for attr 'Tshape' of float is not in the list of allowed values: int32, int64
        ; NodeDef: {{node Reshape}}; Op<name=Reshape; signature=tensor:T, shape:Tshape -> output:T; attr=T:type; attr=Tshape:type,default=DT_INT32,allowed=[DT_INT32, DT_INT64]> [Op:Reshape]

There is also a repository containing Python 3 version of MultiGarment-Network:

minar09/MGN-Py3

Multi-Garment Network implementation for Python3. Contribute to minar09/MGN-Py3 development by creating an account on GitHub.

GitHubminar09