Introduction

Let me introduce my self, Who is Fadhil? Fadhil Riyanto (real-name), is a self-taught system programmer (also backend programmer) which live in Indonesian, This is some short & dirty tutorial about computer science and litte about math.

mirrors: blog.fadhils.eu.org

Contributing

Found a mistake? You can freely DM me on telegram or just email me on

• me@fadev.org (primary, fast respond)
• fadhilriyanto@duck.com (gmail mirror)
• fadhil.riyanto@gnuweeb.org (alternative-1)
• xor@3xploit.dev (alternative-2)

other intresting site

• x86_64 Instruction Set Reference
• some nice book collection (WIP)
• git.fadev.org (WIP)

Language

Because I'm indonesian, some part of this website may written in indonesian. But I will try to translating it into english language.

todo

• MAKE: ./server/php-fpm-config.md book
• COMPLETION: ./database/postgresql_system_table
• COMPLETION: ./database/postgresql_functions
• MAKE: ./crypto/ca-certificates
• MAKE: "freeRADIUS deploy"

System programming

How data movement changes a destination register

This snippets is from "Computer Systems A Programmer’s Perspective, by Randal E. Bryant and David R. O’Hallaron"

As described, there are two different conventions regarding whether and how data movement instruc- tions modify the upper bytes of a destination register. This distinction is illustrated by the following code sequence:

1 movabsq $0x0011223344556677, %rax %rax = 0011223344556677
2 movb $-1, %al %rax = 00112233445566FF
3 movw $-1, %ax %rax = 001122334455FFFF
4 movl $-1, %eax %rax = 00000000FFFFFFFF
5 movq $-1, %rax %rax = FFFFFFFFFFFFFFFF

In the following discussion, we use hexadecimal notation. In the example, the instruction on line 1 initializes register %rax to the pattern 0011223344556677. The remaining instructions have immediate value −1 as their source values. Recall that the hexadecimal representation of −1 is of the form FF. . .F, where the number of F’s is twice the number of bytes in the representation. The movb instruction (line 2) therefore sets the low-order byte of %rax to FF, while the movw instruction (line 3) sets the low-order 2 bytes to FFFF, with the remaining bytes unchanged. The movl instruction (line 4) sets the low-order 4 bytes to FFFFFFFF, but it also sets the high-order 4 bytes to 00000000. Finally, the movq instruction (line 5) sets the complete register to FFFFFFFFFFFFFFFF.

some useful links

• https://stackoverflow.com/questions/52434073/whats-the-difference-between-the-x86-64-att-instructions-movq-and-movabsq

General embedded ASM linked in C

this is program is succesfully compiled with as, but failed to run due to SIGSEGV.

this is will fail

.data
val:
    .byte 0xF6
    .byte 0xF6
    .byte 0xF6
    .byte 0xF6
    

.globl do_asm

do_asm:
    # movl 0x12345678, %eax
    # movl 0xAABBCCDD, %edx 

    # # move edx 8 bit high into al (eax)
    # movb %dh, %al

    
    movq $9, %rax
    ret

this is will compile

.data
val:
    .byte 0xF6
    .byte 0xF6
    .byte 0xF6
    .byte 0xF6
    

.globl do_asm

.text
do_asm:
    # movl 0x12345678, %eax
    # movl 0xAABBCCDD, %edx 

    # # move edx 8 bit high into al (eax)
    # movb %dh, %al

    
    movq $9, %rax
    ret

the difference only in .text section

Move sign & zero-extends a single byte

Disini saya akan membahas apa itu instruksi MOVSBL dan MOVZBL. pertama tama silahkan baca dulu disini

• https://stackoverflow.com/questions/7861095/what-does-movsbl-instruction-do

MOVSBL and MOVZBL
* MOVSBL sign-extends a single byte, and copies it into a
  double-word destination
* MOVZBL expands a single byte to 32 bits with 24 leading
  zeros, and copies it into a double-word destination

pertama tama, lihat kode asm ini.

.section .data
val:
    .byte 0xF6
    .byte 0xF6
    .byte 0xF6
    .byte 0xF6
    

.globl do_asm

.text
do_asm:
    movl $0x12345678, %eax
    movl $0xAABBCCDD, %edx 

    # move edx 8 bit high into al (eax)
    movb %dh, %al

    # reset
    movl $0x12345678, %eax
    movl $0xAABBCCDD, %edx 

    MOVSBL %dh, %eax

    # reset
    movl $0x12345678, %eax
    movl $0xAABBCCDD, %edx 

    MOVZBL %dh, %eax

    
    movq $9, %rax
    ret

dan ini hasilnya, movb biasa (tanpa hapus bagian atas data, bagian atas tetep utuh)

ini hasil signed bagian atas

ini hasil ketika bagian atas kita zerokan

catatan movb %dh, %al memindahkan rdx bagian high ke rax bagian bawah.

movabsq

simplenya, ini kepanjangan dari move absolute quad word (64 bit) contoh kecil

itu contoh initial moveabsq int 64 bit, lalu

contoh move 1 bytes

move 16 bit, 2 bytes word

move 32 bit, 4 bytes, tapi disini keformat sebagai int32 wkwk

full 64 bit move

x86_64 register (fiks technical grade)

analisis kenapa pakai movabsq, daripada movq

movq hanya bisa accept integer

analisis patch movq pakai 0xffffffff

ada asm

kita akan memaksa dari 0x7fffffff menjadi 0xffffffff untuk membuktikan bahwa harusnya movq error, dan digantikan oleh movabsq

initial dulu

lalu, next replace address 0x555555555119 dengan

set {unsigned char[8]}0x555555555119 = {0x48, 0xc7, 0xc0, 0xff, 0xff, 0xff, 0xff, 0x90}

confirmed berubah jadi 0xffffffff disini

saatnya kita lanjutkan, apakah error

ext4 unknown RO compact features

this error come from linux kernel

	/* Check that feature set is OK for a read-write mount */
	if (ext4_has_unknown_ext4_ro_compat_features(sb)) {
		ext4_msg(sb, KERN_ERR, "couldn't mount RDWR because of "
			 "unsupported optional features (%x)",
			 (le32_to_cpu(EXT4_SB(sb)->s_es->s_feature_ro_compat) &
				~EXT4_FEATURE_RO_COMPAT_SUPP));
		return 0;
	}

which EXT4_FEATURE_RO_COMPAT_SUPP

#define EXT4_FEATURE_RO_COMPAT_SUPP	(EXT4_FEATURE_RO_COMPAT_SPARSE_SUPER| \
					 EXT4_FEATURE_RO_COMPAT_LARGE_FILE| \
					 EXT4_FEATURE_RO_COMPAT_GDT_CSUM| \
					 EXT4_FEATURE_RO_COMPAT_DIR_NLINK | \
					 EXT4_FEATURE_RO_COMPAT_EXTRA_ISIZE | \
					 EXT4_FEATURE_RO_COMPAT_BTREE_DIR |\
					 EXT4_FEATURE_RO_COMPAT_HUGE_FILE |\
					 EXT4_FEATURE_RO_COMPAT_BIGALLOC |\
					 EXT4_FEATURE_RO_COMPAT_METADATA_CSUM|\
					 EXT4_FEATURE_RO_COMPAT_QUOTA |\
					 EXT4_FEATURE_RO_COMPAT_PROJECT |\
					 EXT4_FEATURE_RO_COMPAT_VERITY |\
					 EXT4_FEATURE_RO_COMPAT_ORPHAN_PRESENT)

#define EXT4_FEATURE_RO_COMPAT_GDT_CSUM		0x0010

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/ext4/ext4.h#n2188

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/ext4/ext4.h#n2024

qemu custom cpio HDA

skrip init custom linux

qemu-system-x86_64 \-kernel ./vmlinux \
                                                 -nographic \
                                                 --append "console=tty0 console=ttyS0 panic=1 root=/dev/sda rootfstype=ext2" \
                                                 -hda root.img \
                                                 -m 1024 \
                                                 -vga none \
                                                 -display none \
                                                 -serial mon:stdio \
                                                 -no-reboot \
                                                 -initrd vfs/root.cpio.gz

generate cpio

find . | cpio -o -H newc | gzip > root.cpio.gz
``

step TCP

TCP steps

• isi dulu sockaddr_storage, isi pakai inet_pton(); .so_family = AF_INET; sin_port = htons(port)
• syscall socket(AF_INET, SOCK_STREAM, 0);
• setsocksopt(fd, SO_REUSEADDR = 1)
• bind(fd, ss_addr (cast dulu ke sockaddr_in), len nya)
• listen()

penjelasan ttg kenapa pakai sockaddr_storage dipakai https://stackoverflow.com/questions/19528820/difference-between-sockaddr-and-sockaddr-storage

kenapa harus setsocksopt SO_REUSEADDR? karna jika tidak, TCP Masuk ke timewait state.

sekiranya itu aja sih step TCP ini. untuk UDP pakai aja dgram.

GDB attach

jadi ceritanya, ada bug yg kala di run di terminal, ia buggy, pas di run di gdb, dia gak error sama sekali

gw bingung dah, nah disini. ternyata ada trick nya

• jalanin dulu programnya kek biasa di terminal, dah running kan

• ambil pid nya pakai ps aux | grep xyz

• setelah dapet pid nya, jalankan gdb dengan mode attach pid

pakai sudo gdb -p 12345

• program akan freeze sejenak, pencet c, enter

nah, lakukan cara yg dipake buat reproduce bug nya, misal pakai nc buat nyepam tcp ke program. ok dapet bug nya

ni kalau kata orang internet, beberapa bug kek app multi thread agak susah di caught. pakai valgrind pun perlu flags2 khusus buat inspect sampai thread nya

hasil oprek socket programming, case socks5 server

bagian IETF SOCKS5 section 6

https://datatracker.ietf.org/doc/html/rfc1928#section-6

BND.ADDR haruslah berupa IPV4/IPV6 address, bukan domain :)

secara general, bentuk packet yg akan dikirim untuk versi IPV4 seperti ini

[0] [VER] => 0x5
[1] [REP] => 0x0
[2] [RSV] => 0x0
[3] [ATYP] => 0x1 atau 0x4
[4] [BND.ADDR] 0xac
[5] [BND.ADDR] 0x43
[6] [BND.ADDR] 0x93
[7] [BND.ADDR] 0x39
[8] [BND.PORT] 0x50 
[9] [BND.PORT] 0x50

maka perhitungannya memory awal + 4 (karna 4 field dah dipakai), start dari sini, insert 4 oktet lagi ipv4 address, lalu insert 2 oktet uint8 sebagai port.

bagian BND.PORT itu unsigned int 16 bit. jadi harus digabung bitnya + diubah dari network order ke host order

fix ncurses kernel build

suppose

*** Unable to find the ncurses libraries or the
 *** required header files.
 *** 'make menuconfig' requires the ncurses libraries.
 *** 
 *** Install ncurses (ncurses-devel) and try again.
 *** 
make[2]: *** [/home/fadhil_riyanto/linux-kernel/busybox-1.37.0/scripts/kconfig/lxdialog/Makefile:15: scripts/kconfig/lxdialog/dochecklxdialog] Error 1
make[1]: *** [/home/fadhil_riyanto/linux-kernel/busybox-1.37.0/scripts/kconfig/Makefile:14: menuconfig] Error 2
make: *** [Makefile:444: menuconfig] Error 2

maka solusi nano scripts/kconfig/lxdialog/Makefile

ubah

always         := $(hostprogs-y) dochecklxdialog

PHP-SRC variadic params

the source code: https://github.com/php/php-src/blob/ba6c00505d4218ff9fe4feb060f636524acfb125/ext/standard/array.c#L1204

mayan ngebantu jg gw ngepahami variadic params di php. ref: https://www.zend.com/resources/php-extensions/basic-php-structures

selfnote :

ZEND_PARSE_PARAMETERS_START(num_required_args, num_max_args) Z_PARAM_VARIADIC('*' or '+', destptr, argc);

catatan:

• * nol atau lebih parameter
• + satu atau lebih parameter

nb: akan diupdate berkala

ref: https://wiki.php.net/rfc/fast_zpp

PHP-SRC minit()

https://github.com/php/php-src/blob/3704947696fe0ee93e025fa85621d297ac7a1e4d/main/main.c#L2009

Ketika module PHP diload ke zend engine, yang dipanggil adalah MINIT(), di step ini yg terjadi cuman memory alokasi di ZendMM, dan disini engga ada process atau thread yg dijalan, pure cuman initialisasi data

nah, yang mentrigger MINIT() adalah zend_startup_modules(), trus secara beurutan, ia jg manggil RINIT() via PHP_RINIT_FUNCTION()

struct nya zend_result (*request_startup_func)(INIT_FUNC_ARGS);

gcc compiler visibility

intresting buat diulik

class rectangle {
        int width, height;

        public:
                void set_value(int, int);
                int area();
};

void rectangle::set_value(int x, int y) 
{
        width = x;
        height = y;
}


int rectangle::area()
{
        return width * height;
}

compile pakai

• g++ -Wall -c -fPIC abc.cc
• g++ -o librec.so --shared abc.o
• nm -C librec.so

akan ada

000000000000110e T rectangle::area()
00000000000010ea T rectangle::set_value(int, int)

nah, disitu kalau di compile pakai -fvisibility=hidden ntar pas di cek pakai nm -CD librec.so

bakalan gak ada. nah ini ada kaitannya sama __attribute__ ((visibility ("hidden")))

tambahan: https://groups.google.com/g/ode-users/c/T3qiy-4dviQ

load effective address

computes a memory address using the same arithmetic that a MOV instruction uses. But unlike the MOV instruction, the LEA instruction just stores the computed address in its target register, instead of loading the contents of that address and storing it.

Consider your first LEA instruction:

tldr nya: mov bisa mengkalkulasi operasi aritmetika dari alamat memori, trus ngambil isinya kalau lea, ia cuman mengkalkulasi alamatnya, hasil itung2an nya disimpan ke operand2

lea -0x18(%ebp), %ebx

tambahan catatan

mov (op1, op2, op3), dest

equal (pseudo): mov op1 + op2 * op3, dest

and lea op1(op2), dest

equal lea op2 + op1, dest (address of op2 + op1, store result into dest)

bahasan tentang rip https://stackoverflow.com/questions/54745872/how-do-rip-relative-variable-references-like-rip-a-in-x86-64-gas-intel-sy

AT&T asm prelude

push eax equal with: mov [esp], eax sub esp, 4

pop eax get values out of stack equal with

add rsp, 4 mov eax, [esp]

printf format specifier

general

%[flags][width][.precision][length]specifier

flag

width

The width sub-specifier sets the minimum number of characters to be printed.

precision

length

specifier

predefined macros

links: https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html clang spesific: https://clang.llvm.org/docs/LanguageExtensions.html

vt-hexdump

/* SPDX-License-Identifier: GPL-2.0-only */
/*
 * Copyright (C) Fadhil Riyanto <fadhil.riyanto@gnuweeb.org>
 *
 * this version is inspired by Ammar Faizi's versions
 * ref: https://gist.githubusercontent.com/ammarfaizi2/e88a21171358b5092a3df412eeb80b2f/raw/59141b48f59b70b1e96208716c45b1703c56daa7/vt_hexdump.h
 */

#include <complex.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>

#ifndef VT_HEXDUMP_COLOR
#define VT_HEXDUMP_COLOR(fmt, vt_hexdump_hex)                                   \
        char current_hex = (char)vt_hexdump_hex;                                \
                                                                                \
        if (current_hex == 0x7F) {                                              \
                printf("\033[1;31m" fmt "\033[0m", vt_hexdump_hex);             \
        } else if (current_hex == 0xFF) {                                       \
                printf("\033[1;34m" fmt "\033[0m", vt_hexdump_hex);             \
        } else if (current_hex == 0x00) {                                       \
                printf("\033[1;37m" fmt "\033[0m", vt_hexdump_hex);             \
        } else {                                                                \
                printf("\033[1;32m" fmt "\033[0m", vt_hexdump_hex);             \
        }                                                                       \
        
#endif /* VT_HEXDUMP_COLOR */

#define VT_TITLE(PTR, SIZE)                                                     \
        size_t t_ptr_size = SIZE;                                                 \
        unsigned char *t_realptr = (unsigned char*)PTR;                           \
        printf("================= VT_HEXDUMP =================\n");             \
        printf("file\t\t: %s:%d\n", __FILE__, __LINE__);                        \
        printf("func\t\t: %s\n", __FUNCTION__);                                 \
        printf("addr\t\t: 0x%016lx\n", t_realptr);                                \
        printf("dump_size\t: %ld\n\n", t_ptr_size);

#ifndef HEXDUMP
#define HEXDUMP(PTR, SIZE)                                                      \
        size_t ptr_size = SIZE;                                                 \
        unsigned char *realptr = (unsigned char*)PTR;                           \
        unsigned int initial_counter = 0;                                       \
                                                                                \
        int n_loop = (ptr_size / 16) + 1;                                       \
        if (ptr_size % 16 == 0) {                                               \
                n_loop = n_loop - 1;                                            \
        }                                                                       \
                                                                                \
        for (int x = 0; x < 75; x++) {                                          \
                if (x >= 40) {                                                  \
                        printf("16 BYTES WIDE\n");                              \
                        break;                                                  \
                } else {                                                        \
                        printf(" ");                                            \
                }                                                               \
        }                                                                       \
                                                                                \
        for (int x = 0; x < 75; x++) {                                          \
                if (x >= 21 && x <= 73) {                                       \
                        printf("_");                                            \
                } else {                                                        \
                        printf(" ");                                            \
                }                                                               \
        }                                                                       \
        printf("\n");                                                           \
                                                                                \
        for (int i = 0; i < n_loop; i++) {                                      \
                printf("|0x%016lx|", (uintptr_t)(realptr));                     \
                                                                                \
                for (int i = 0; i < 16; i++) {                                  \
                        if (i % 4 == 0 && i != 0) {                             \
                                printf("  ");                                   \
                        }                                                       \
                        if (i != 15) {                                          \
                                if (initial_counter <= ptr_size) {              \
                                        VT_HEXDUMP_COLOR(" %02x", realptr[i]);  \
                                }                                               \
                        } else {                                                \
                                printf(" %02x ", realptr[i]);                   \
                        }                                                       \
                }                                                               \
                                                                                \
                printf(" | ");                                                  \
                for (int i = 0; i < 16; i++) {                                  \
                        if (initial_counter <= ptr_size - 1) {                  \
                                if (realptr[i] >= 32 && realptr[i] <= 126) {    \
                                        printf("%c", realptr[i]);               \
                                } else {                                        \
                                        printf(".", realptr[i]);                \
                                }                                               \
                        } else {                                                \
                                printf(".");                                    \
                        }                                                       \
                        initial_counter = initial_counter + 1;                  \
                }                                                               \
                printf(" | ");                                                  \
                                                                                \
                printf("\n");                                                   \
                realptr = realptr + 16;                                         \
        }
#endif                                     

#ifndef VT_HEXDUMP
#define VT_HEXDUMP(PTR, SIZE)                                                   \
        VT_TITLE(PTR, SIZE);                                                             \
        HEXDUMP(PTR, SIZE);
#endif

ELF64 text only

ELF Types

There are three main types for ELF files.

• An executable file contains code and data suitable for execution. It specifies the memory layout of the process.
• A relocatable file contains code and data suitable for linking with other relocatable and shared object files.
• A shared object file (a.k.a. shared library) contains code and data suitable for the link editor ld at link time and the dynamic linker at run time. The dynamic linker may be called ld.so.1, libc.so.1 or ld-linux.so.1, depending on the implementation.

The most useful part of ELF lies in its section structure. With the right tools and techniques, programmers can manipulate the execution of executables with great flexibility.

source: https://ftp.math.utah.edu/u/ma/hohn/linux/misc/elf/node2.html

big fat ELF docs: https://docs.oracle.com/cd/E19683-01/816-1386/6m7qcoblh/index.html, big thanks to oracle!

ELF64

what is ELF64? ELF64 is Executable and Linkable Format for unix system (mostly), lets look the structure of elf64_hdr

this struct located at beginning of object file, the main usage is locate all other parts of object file. the header (such *.o file contains ELF struct)

preface

this is how ELF file look-like

typedef struct elf64_hdr {
  unsigned char	e_ident[EI_NIDENT];	/* ELF "magic number" */
  Elf64_Half e_type;
  Elf64_Half e_machine;
  Elf64_Word e_version;
  Elf64_Addr e_entry;		/* Entry point virtual address */
  Elf64_Off e_phoff;		/* Program header table file offset */
  Elf64_Off e_shoff;		/* Section header table file offset */
  Elf64_Word e_flags;
  Elf64_Half e_ehsize;
  Elf64_Half e_phentsize;
  Elf64_Half e_phnum;
  Elf64_Half e_shentsize;
  Elf64_Half e_shnum;
  Elf64_Half e_shstrndx;
} Elf64_Ehdr;

code: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/elf.h#n234

the ELF file is always start by this bytes 7f 45 4c 46, which DEL E L F

The first 16 bytes link is a magic number, 7f make sure the text editor, this is not a normal file, this is binary, preventing to open the file. 45 4c 46 is a ELF signature

ELFCLASS

This footage shown program compiled by normal gcc (my machine is 64 bit), vs compiled by -m32

ELF file structure

• elf64_hdr: elf header
• elf64_shdr: elf section header
• elf64_phdr: elf program header

elf64_hdr

the definition

typedef struct elf64_hdr {
  unsigned char e_ident[EI_NIDENT]; /* Magic number and other info */
  Elf64_Half    e_type;      /* Object file type */
  Elf64_Half    e_machine;   /* Architecture */
  Elf64_Word    e_version;   /* Object file version */
  Elf64_Addr    e_entry;     /* Entry point virtual address */
  Elf64_Off     e_phoff;     /* Program header table file offset */
  Elf64_Off     e_shoff;     /* Section header table file offset */
  Elf64_Word    e_flags;     /* Processor-specific flags */
  Elf64_Half    e_ehsize;    /* ELF header size in bytes */
  Elf64_Half    e_phentsize; /* Program header table entry size */
  Elf64_Half    e_phnum;     /* Program header table entry count */
  Elf64_Half    e_shentsize; /* Section header table entry size */
  Elf64_Half    e_shnum;     /* Section header table entry count */
  Elf64_Half    e_shstrndx;  /* Section names string table index */
} Elf64_Ehdr;

link go to the definition

• e_ident: Magic number and other info
• e_type: Object file type
• e_machine: Architecture
• e_version: Object file version
• e_entry: Entry point virtual address
• e_phoff: Program header table file offset
• e_shoff: Section header table file offset
• e_flags: Processor-specific flags
• e_ehsize: ELF header size in bytes
• e_phentsize: Program header table entry size
• e_phnum: Program header table entry count
• e_shentsize: Section header table entry size
• e_shnum: Section header table entry count
• e_shstrndx: Section names string table index

the value (enum like)

e_ident

• INDEX: [0, 1, 2, 3]

  #define EI_MAG0         0               /* File identification byte 0 index */
  #define ELFMAG0         0x7f            /* Magic number byte 0 */

  #define EI_MAG1         1               /* File identification byte 1 index */
  #define ELFMAG1         'E'             /* Magic number byte 1 */

  #define EI_MAG2         2               /* File identification byte 2 index */
  #define ELFMAG2         'L'             /* Magic number byte 2 */

  #define EI_MAG3         3               /* File identification byte 3 index */
  #define ELFMAG3         'F'             /* Magic number byte 3 */

• INDEX: [4]

#define ELFCLASSNONE    0               /* Invalid class */
#define ELFCLASS32      1               /* 32-bit objects */
#define ELFCLASS64      2               /* 64-bit objects */
#define ELFCLASSNUM     3

• INDEX: [5]

#define ELFDATANONE     0               /* Invalid data encoding */
#define ELFDATA2LSB     1               /* 2's complement, little endian */
#define ELFDATA2MSB     2               /* 2's complement, big endian */
#define ELFDATANUM      3

• INDEX: [6]

#define EI_VERSION      6               /* File version byte index */
                                        /* Value must be EV_CURRENT */

• INDEX: [7]

#define ELFOSABI_NONE           0       /* UNIX System V ABI */
#define ELFOSABI_SYSV           0       /* Alias.  */
#define ELFOSABI_HPUX           1       /* HP-UX */
#define ELFOSABI_NETBSD         2       /* NetBSD.  */
#define ELFOSABI_GNU            3       /* Object uses GNU ELF extensions.  */
#define ELFOSABI_LINUX          ELFOSABI_GNU /* Compatibility alias.  */
#define ELFOSABI_SOLARIS        6       /* Sun Solaris.  */
#define ELFOSABI_AIX            7       /* IBM AIX.  */
#define ELFOSABI_IRIX           8       /* SGI Irix.  */
#define ELFOSABI_FREEBSD        9       /* FreeBSD.  */
#define ELFOSABI_TRU64          10      /* Compaq TRU64 UNIX.  */
#define ELFOSABI_MODESTO        11      /* Novell Modesto.  */
#define ELFOSABI_OPENBSD        12      /* OpenBSD.  */
#define ELFOSABI_ARM_AEABI      64      /* ARM EABI */
#define ELFOSABI_ARM            97      /* ARM */
#define ELFOSABI_STANDALONE     255     /* Standalone (embedded) application */

• INDEX: [8]

#define EI_ABIVERSION   8               /* ABI version */

• INDEX: [9]

#define EI_PAD          9               /* Byte index of padding bytes */

back

e_type

#define ET_NONE         0               /* No file type */
#define ET_REL          1               /* Relocatable file */
#define ET_EXEC         2               /* Executable file */
#define ET_DYN          3               /* Shared object file */
#define ET_CORE         4               /* Core file */
#define ET_NUM          5               /* Number of defined types */
#define ET_LOOS         0xfe00          /* OS-specific range start */
#define ET_HIOS         0xfeff          /* OS-specific range end */
#define ET_LOPROC       0xff00          /* Processor-specific range start */
#define ET_HIPROC       0xffff          /* Processor-specific range end */

back

e_machine

#define EM_NONE    0  /* No machine */
#define EM_M32     1  /* AT&T WE 32100 */
#define EM_SPARC   2  /* SUN SPARC */
#define EM_386     3  /* Intel 80386 */
#define EM_68K     4  /* Motorola m68k family */
#define EM_88K     5  /* Motorola m88k family */
#define EM_IAMCU   6  /* Intel MCU */
#define EM_860     7  /* Intel 80860 */
#define EM_MIPS    8  /* MIPS R3000 big-endian */
#define EM_S370    9  /* IBM System/370 */
#define EM_MIPS_RS3_LE  10  /* MIPS R3000 little-endian */
        /* reserved 11-14 */
#define EM_PARISC 15  /* HPPA */
        /* reserved 16 */
#define EM_VPP500 17  /* Fujitsu VPP500 */
#define EM_SPARC32PLUS  18  /* Sun's "v8plus" */
#define EM_960    19  /* Intel 80960 */
#define EM_PPC    20  /* PowerPC */
#define EM_PPC64  21  /* PowerPC 64-bit */
#define EM_S390   22  /* IBM S390 */
#define EM_SPU    23  /* IBM SPU/SPC */
        /* reserved 24-35 */
#define EM_V800   36  /* NEC V800 series */
#define EM_FR20   37  /* Fujitsu FR20 */
#define EM_RH32   38  /* TRW RH-32 */
#define EM_RCE    39  /* Motorola RCE */
#define EM_ARM    40  /* ARM */
#define EM_FAKE_ALPHA 41  /* Digital Alpha */
#define EM_SH   42  /* Hitachi SH */
#define EM_SPARCV9  43  /* SPARC v9 64-bit */
#define EM_TRICORE  44  /* Siemens Tricore */
#define EM_ARC    45  /* Argonaut RISC Core */
#define EM_H8_300 46  /* Hitachi H8/300 */
#define EM_H8_300H  47  /* Hitachi H8/300H */
#define EM_H8S    48  /* Hitachi H8S */
#define EM_H8_500 49  /* Hitachi H8/500 */
#define EM_IA_64  50  /* Intel Merced */
#define EM_MIPS_X 51  /* Stanford MIPS-X */
#define EM_COLDFIRE 52  /* Motorola Coldfire */
#define EM_68HC12 53  /* Motorola M68HC12 */
#define EM_MMA    54  /* Fujitsu MMA Multimedia Accelerator */
#define EM_PCP    55  /* Siemens PCP */
#define EM_NCPU   56  /* Sony nCPU embedded RISC */
#define EM_NDR1   57  /* Denso NDR1 microprocessor */
#define EM_STARCORE 58  /* Motorola Start*Core processor */
#define EM_ME16   59  /* Toyota ME16 processor */
#define EM_ST100  60  /* STMicroelectronic ST100 processor */
#define EM_TINYJ  61  /* Advanced Logic Corp. Tinyj emb.fam */
#define EM_X86_64 62  /* AMD x86-64 architecture */
#define EM_PDSP   63  /* Sony DSP Processor */
#define EM_PDP10  64  /* Digital PDP-10 */
#define EM_PDP11  65  /* Digital PDP-11 */
#define EM_FX66   66  /* Siemens FX66 microcontroller */
#define EM_ST9PLUS  67  /* STMicroelectronics ST9+ 8/16 mc */
#define EM_ST7    68  /* STmicroelectronics ST7 8 bit mc */
#define EM_68HC16 69  /* Motorola MC68HC16 microcontroller */
#define EM_68HC11 70  /* Motorola MC68HC11 microcontroller */
#define EM_68HC08 71  /* Motorola MC68HC08 microcontroller */
#define EM_68HC05 72  /* Motorola MC68HC05 microcontroller */
#define EM_SVX    73  /* Silicon Graphics SVx */
#define EM_ST19   74  /* STMicroelectronics ST19 8 bit mc */
#define EM_VAX    75  /* Digital VAX */
#define EM_CRIS   76  /* Axis Communications 32-bit emb.proc */
#define EM_JAVELIN  77  /* Infineon Technologies 32-bit emb.proc */
#define EM_FIREPATH 78  /* Element 14 64-bit DSP Processor */
#define EM_ZSP    79  /* LSI Logic 16-bit DSP Processor */
#define EM_MMIX   80  /* Donald Knuth's educational 64-bit proc */
#define EM_HUANY  81  /* Harvard University machine-independent object files */
#define EM_PRISM  82  /* SiTera Prism */
#define EM_AVR    83  /* Atmel AVR 8-bit microcontroller */
#define EM_FR30   84  /* Fujitsu FR30 */
#define EM_D10V   85  /* Mitsubishi D10V */
#define EM_D30V   86  /* Mitsubishi D30V */
#define EM_V850   87  /* NEC v850 */
#define EM_M32R   88  /* Mitsubishi M32R */
#define EM_MN10300  89  /* Matsushita MN10300 */
#define EM_MN10200  90  /* Matsushita MN10200 */
#define EM_PJ   91  /* picoJava */
#define EM_OPENRISC 92  /* OpenRISC 32-bit embedded processor */
#define EM_ARC_COMPACT  93  /* ARC International ARCompact */
#define EM_XTENSA 94  /* Tensilica Xtensa Architecture */
#define EM_VIDEOCORE  95  /* Alphamosaic VideoCore */
#define EM_TMM_GPP  96  /* Thompson Multimedia General Purpose Proc */
#define EM_NS32K  97  /* National Semi. 32000 */
#define EM_TPC    98  /* Tenor Network TPC */
#define EM_SNP1K  99  /* Trebia SNP 1000 */
#define EM_ST200  100 /* STMicroelectronics ST200 */
#define EM_IP2K   101 /* Ubicom IP2xxx */
#define EM_MAX    102 /* MAX processor */
#define EM_CR   103 /* National Semi. CompactRISC */
#define EM_F2MC16 104 /* Fujitsu F2MC16 */
#define EM_MSP430 105 /* Texas Instruments msp430 */
#define EM_BLACKFIN 106 /* Analog Devices Blackfin DSP */
#define EM_SE_C33 107 /* Seiko Epson S1C33 family */
#define EM_SEP    108 /* Sharp embedded microprocessor */
#define EM_ARCA   109 /* Arca RISC */
#define EM_UNICORE  110 /* PKU-Unity & MPRC Peking Uni. mc series */
#define EM_EXCESS 111 /* eXcess configurable cpu */
#define EM_DXP    112 /* Icera Semi. Deep Execution Processor */
#define EM_ALTERA_NIOS2 113 /* Altera Nios II */
#define EM_CRX    114 /* National Semi. CompactRISC CRX */
#define EM_XGATE  115 /* Motorola XGATE */
#define EM_C166   116 /* Infineon C16x/XC16x */
#define EM_M16C   117 /* Renesas M16C */
#define EM_DSPIC30F 118 /* Microchip Technology dsPIC30F */
#define EM_CE   119 /* Freescale Communication Engine RISC */
#define EM_M32C   120 /* Renesas M32C */
        /* reserved 121-130 */
#define EM_TSK3000  131 /* Altium TSK3000 */
#define EM_RS08   132 /* Freescale RS08 */
#define EM_SHARC  133 /* Analog Devices SHARC family */
#define EM_ECOG2  134 /* Cyan Technology eCOG2 */
#define EM_SCORE7 135 /* Sunplus S+core7 RISC */
#define EM_DSP24  136 /* New Japan Radio (NJR) 24-bit DSP */
#define EM_VIDEOCORE3 137 /* Broadcom VideoCore III */
#define EM_LATTICEMICO32 138  /* RISC for Lattice FPGA */
#define EM_SE_C17 139 /* Seiko Epson C17 */
#define EM_TI_C6000 140 /* Texas Instruments TMS320C6000 DSP */
#define EM_TI_C2000 141 /* Texas Instruments TMS320C2000 DSP */
#define EM_TI_C5500 142 /* Texas Instruments TMS320C55x DSP */
#define EM_TI_ARP32 143 /* Texas Instruments App. Specific RISC */
#define EM_TI_PRU 144 /* Texas Instruments Prog. Realtime Unit */
        /* reserved 145-159 */
#define EM_MMDSP_PLUS 160 /* STMicroelectronics 64bit VLIW DSP */
#define EM_CYPRESS_M8C  161 /* Cypress M8C */
#define EM_R32C   162 /* Renesas R32C */
#define EM_TRIMEDIA 163 /* NXP Semi. TriMedia */
#define EM_QDSP6  164 /* QUALCOMM DSP6 */
#define EM_8051   165 /* Intel 8051 and variants */
#define EM_STXP7X 166 /* STMicroelectronics STxP7x */
#define EM_NDS32  167 /* Andes Tech. compact code emb. RISC */
#define EM_ECOG1X 168 /* Cyan Technology eCOG1X */
#define EM_MAXQ30 169 /* Dallas Semi. MAXQ30 mc */
#define EM_XIMO16 170 /* New Japan Radio (NJR) 16-bit DSP */
#define EM_MANIK  171 /* M2000 Reconfigurable RISC */
#define EM_CRAYNV2  172 /* Cray NV2 vector architecture */
#define EM_RX   173 /* Renesas RX */
#define EM_METAG  174 /* Imagination Tech. META */
#define EM_MCST_ELBRUS  175 /* MCST Elbrus */
#define EM_ECOG16 176 /* Cyan Technology eCOG16 */
#define EM_CR16   177 /* National Semi. CompactRISC CR16 */
#define EM_ETPU   178 /* Freescale Extended Time Processing Unit */
#define EM_SLE9X  179 /* Infineon Tech. SLE9X */
#define EM_L10M   180 /* Intel L10M */
#define EM_K10M   181 /* Intel K10M */
        /* reserved 182 */
#define EM_AARCH64  183 /* ARM AARCH64 */
        /* reserved 184 */
#define EM_AVR32  185 /* Amtel 32-bit microprocessor */
#define EM_STM8   186 /* STMicroelectronics STM8 */
#define EM_TILE64 187 /* Tilera TILE64 */
#define EM_TILEPRO  188 /* Tilera TILEPro */
#define EM_MICROBLAZE 189 /* Xilinx MicroBlaze */
#define EM_CUDA   190 /* NVIDIA CUDA */
#define EM_TILEGX 191 /* Tilera TILE-Gx */
#define EM_CLOUDSHIELD  192 /* CloudShield */
#define EM_COREA_1ST  193 /* KIPO-KAIST Core-A 1st gen. */
#define EM_COREA_2ND  194 /* KIPO-KAIST Core-A 2nd gen. */
#define EM_ARCV2  195 /* Synopsys ARCv2 ISA.  */
#define EM_OPEN8  196 /* Open8 RISC */
#define EM_RL78   197 /* Renesas RL78 */
#define EM_VIDEOCORE5 198 /* Broadcom VideoCore V */
#define EM_78KOR  199 /* Renesas 78KOR */
#define EM_56800EX  200 /* Freescale 56800EX DSC */
#define EM_BA1    201 /* Beyond BA1 */
#define EM_BA2    202 /* Beyond BA2 */
#define EM_XCORE  203 /* XMOS xCORE */
#define EM_MCHP_PIC 204 /* Microchip 8-bit PIC(r) */
#define EM_INTELGT  205 /* Intel Graphics Technology */
        /* reserved 206-209 */
#define EM_KM32   210 /* KM211 KM32 */
#define EM_KMX32  211 /* KM211 KMX32 */
#define EM_EMX16  212 /* KM211 KMX16 */
#define EM_EMX8   213 /* KM211 KMX8 */
#define EM_KVARC  214 /* KM211 KVARC */
#define EM_CDP    215 /* Paneve CDP */
#define EM_COGE   216 /* Cognitive Smart Memory Processor */
#define EM_COOL   217 /* Bluechip CoolEngine */
#define EM_NORC   218 /* Nanoradio Optimized RISC */
#define EM_CSR_KALIMBA  219 /* CSR Kalimba */
#define EM_Z80    220 /* Zilog Z80 */
#define EM_VISIUM 221 /* Controls and Data Services VISIUMcore */
#define EM_FT32   222 /* FTDI Chip FT32 */
#define EM_MOXIE  223 /* Moxie processor */
#define EM_AMDGPU 224 /* AMD GPU */
        /* reserved 225-242 */
#define EM_RISCV  243 /* RISC-V */

#define EM_BPF    247 /* Linux BPF -- in-kernel virtual machine */
#define EM_CSKY   252     /* C-SKY */
#define EM_LOONGARCH  258 /* LoongArch */

#define EM_NUM    259

/* Old spellings/synonyms.  */

#define EM_ARC_A5 EM_ARC_COMPACT

/* If it is necessary to assign new unofficial EM_* values, please
   pick large random numbers (0x8523, 0xa7f2, etc.) to minimize the
   chances of collision with official or non-GNU unofficial values.  */

#define EM_ALPHA  0x9026

back

e_version

#define EV_NONE         0               /* Invalid ELF version */
#define EV_CURRENT      1               /* Current version */
#define EV_NUM          2

back

elf64_shdr

All data stores in a sections in an Elf object file. Sections identified by index in the section header table.

typedef struct
{
  Elf64_Word    sh_name;                /* Section name (string tbl index) */
  Elf64_Word    sh_type;                /* Section type */
  Elf64_Xword   sh_flags;               /* Section flags */
  Elf64_Addr    sh_addr;                /* Section virtual addr at execution */
  Elf64_Off     sh_offset;              /* Section file offset */
  Elf64_Xword   sh_size;                /* Section size in bytes */
  Elf64_Word    sh_link;                /* Link to another section */
  Elf64_Word    sh_info;                /* Additional section information */
  Elf64_Xword   sh_addralign;           /* Section alignment */
  Elf64_Xword   sh_entsize;             /* Entry size if section holds table */
} Elf64_Shdr;

POSIX getopt_long

optarg

• : for required arguments, for example

  • f: expands to -f abc
  • no : treated only as args, example -v

• :: treated as optional args

usage

read man 3 getopt

tldr

lets construct array of option struct

definition:

struct option {
	const char *name;				// param name
	int         has_arg;			// set 1 or 0 (no)
	int        *flag;				// always 0
	int         val;				// whatever, 1 byte, used for switch block identification in future
};

example:

static struct option long_options[] = {
    {"file", 1, 0, 'f'},
    {"elf", 1, 0, 'e'},
    NULL
};

example

static int parse_options(int argc, char *argv[], struct gwdown_ctx *ctx)
{
	int ret = 0;

	while (1) {
		int c;

		c = getopt_long(argc, argv, "hvo:t:rVM", long_options, NULL);
		if (c == -1)
			break;

		switch (c) {
		case 'h':
			help(argv[0]);
			ret = 255;
			goto out;
		case 'v':
			printf("gwdown 0.1\n");
			ret = 255;
			goto out;
		case 'o':
			ctx->output = optarg;
			break;
		case 't':
			ctx->num_threads = (uint16_t)atoi(optarg);
			break;
		case 'r':
			ctx->resume = true;
			break;
		case 'V':
			ctx->verbose = true;
			break;
		case 'M':
			ctx->use_mmap = true;
			break;
		default:
			printf("Error: Unknown option '%s'\n\n", argv[optind]);
			help(argv[0]);
			ret = -EINVAL;
			goto out;
		}
	}

	if (ctx->num_threads > 512) {
		printf("Error: Too many threads, max allowed thread is 512\n\n");
		ret = -EINVAL;
		goto out;
	}

	if (optind < argc) {
		ctx->url = strdup(argv[optind]);
		if (!ctx->url)
			ret = -ENOMEM;
	} else {
		printf("Error: Missing url argument\n\n");
		help(argv[0]);
		ret = -EINVAL;
	}

	if (ctx->resume) {
		printf("Error: The resume feature is currently not supproted\n");
		printf("       It's still just a draft feature\n\n");
		ret = -EOPNOTSUPP;
	}

out:
	return ret;
}

as you can see, optarg is magically return current character based on switch case cond

POSIX getopt (only)

its parse command line like this

no args params

this option did not receive any arguments, so the optarg value is will be null ./prog -v, use optstring something like this v

optional params

./prog -p=8000 -l=127.0.0.1, use optsting p::l::

this is wrong example

this is good example

required params

./prog -p 8000 -l 0.0.0.0 use optstring p:l:

unknown option handling

the handling is very easy, getopt() return ? when undefined rule is provided, then you can use optopt as a handler

useful links

• https://www.gnu.org/software/libc/manual/html_node/Example-of-Getopt.html

linux virtual memory map

Documentation/x86/x86_64/mm.txt

<previous description obsolete, deleted>

Virtual memory map with 4 level page tables:

0000000000000000 - 00007fffffffffff (=47 bits) user space, different per mm
hole caused by [47:63] sign extension
ffff800000000000 - ffff87ffffffffff (=43 bits) guard hole, reserved for hypervisor
ffff880000000000 - ffffc7ffffffffff (=64 TB) direct mapping of all phys. memory
ffffc80000000000 - ffffc8ffffffffff (=40 bits) hole
ffffc90000000000 - ffffe8ffffffffff (=45 bits) vmalloc/ioremap space
ffffe90000000000 - ffffe9ffffffffff (=40 bits) hole
ffffea0000000000 - ffffeaffffffffff (=40 bits) virtual memory map (1TB)
... unused hole ...
ffffec0000000000 - fffffbffffffffff (=44 bits) kasan shadow memory (16TB)
... unused hole ...
ffffff0000000000 - ffffff7fffffffff (=39 bits) %esp fixup stacks
... unused hole ...
ffffffef00000000 - fffffffeffffffff (=64 GB) EFI region mapping space
... unused hole ...
ffffffff80000000 - ffffffff9fffffff (=512 MB)  kernel text mapping, from phys 0
ffffffffa0000000 - ffffffffff5fffff (=1526 MB) module mapping space (variable)
ffffffffff600000 - ffffffffffdfffff (=8 MB) vsyscalls
ffffffffffe00000 - ffffffffffffffff (=2 MB) unused hole

Virtual memory map with 5 level page tables:

0000000000000000 - 00ffffffffffffff (=56 bits) user space, different per mm
hole caused by [56:63] sign extension
ff00000000000000 - ff0fffffffffffff (=52 bits) guard hole, reserved for hypervisor
ff10000000000000 - ff8fffffffffffff (=55 bits) direct mapping of all phys. memory
ff90000000000000 - ff91ffffffffffff (=49 bits) hole
ff92000000000000 - ffd1ffffffffffff (=54 bits) vmalloc/ioremap space
ffd2000000000000 - ffd3ffffffffffff (=49 bits) hole
ffd4000000000000 - ffd5ffffffffffff (=49 bits) virtual memory map (512TB)
... unused hole ...
ffd8000000000000 - fff7ffffffffffff (=53 bits) kasan shadow memory (8PB)
... unused hole ...
ffffff0000000000 - ffffff7fffffffff (=39 bits) %esp fixup stacks
... unused hole ...
ffffffef00000000 - fffffffeffffffff (=64 GB) EFI region mapping space
... unused hole ...
ffffffff80000000 - ffffffff9fffffff (=512 MB)  kernel text mapping, from phys 0
ffffffffa0000000 - ffffffffff5fffff (=1526 MB) module mapping space
ffffffffff600000 - ffffffffffdfffff (=8 MB) vsyscalls
ffffffffffe00000 - ffffffffffffffff (=2 MB) unused hole

Architecture defines a 64-bit virtual address. Implementations can support
less. Currently supported are 48- and 57-bit virtual addresses. Bits 63
through to the most-significant implemented bit are set to either all ones
or all zero. This causes hole between user space and kernel addresses.

The direct mapping covers all memory in the system up to the highest
memory address (this means in some cases it can also include PCI memory
holes).

vmalloc space is lazily synchronized into the different PML4/PML5 pages of
the processes using the page fault handler, with init_top_pgt as
reference.

Current X86-64 implementations support up to 46 bits of address space (64 TB),
which is our current limit. This expands into MBZ space in the page tables.

We map EFI runtime services in the 'efi_pgd' PGD in a 64Gb large virtual
memory window (this size is arbitrary, it can be raised later if needed).
The mappings are not part of any other kernel PGD and are only available
during EFI runtime calls.

The module mapping space size changes based on the CONFIG requirements for the
following fixmap section.

Note that if CONFIG_RANDOMIZE_MEMORY is enabled, the direct mapping of all
physical memory, vmalloc/ioremap space and virtual memory map are randomized.
Their order is preserved but their base will be offset early at boot time.

-Andi Kleen, Jul 2004

linux x86_64 memory map

========================================================================================================================
    Start addr    |   Offset   |     End addr     |  Size   | VM area description
========================================================================================================================
                  |            |                  |         |
 0000000000000000 |    0       | 00fffffffffff000 |  ~64 PB | user-space virtual memory, different per mm
 00fffffffffff000 |  ~64    PB | 00ffffffffffffff |    4 kB | ... guard hole
__________________|____________|__________________|_________|___________________________________________________________
                  |            |                  |         |
 0100000000000000 |  +64    PB | 7fffffffffffffff |   ~8 EB | ... huge, almost 63 bits wide hole of non-canonical
                  |            |                  |         |     virtual memory addresses up to the -8EB TB
                  |            |                  |         |     starting offset of kernel mappings.
                  |            |                  |         |
                  |            |                  |         | LAM relaxes canonicallity check allowing to create aliases
                  |            |                  |         | for userspace memory here.
__________________|____________|__________________|_________|___________________________________________________________
                                                            |
                                                            | Kernel-space virtual memory, shared between all processes:
____________________________________________________________|___________________________________________________________
 8000000000000000 |   -8    EB | feffffffffffffff |   ~8 EB | ... huge, almost 63 bits wide hole of non-canonical
                  |            |                  |         |     virtual memory addresses up to the -64 PB
                  |            |                  |         |     starting offset of kernel mappings.
                  |            |                  |         |
                  |            |                  |         | LAM_SUP relaxes canonicallity check allowing to create
                  |            |                  |         | aliases for kernel memory here.
____________________________________________________________|___________________________________________________________
                  |            |                  |         |
 ff00000000000000 |  -64    PB | ff0fffffffffffff |    4 PB | ... guard hole, also reserved for hypervisor
 ff10000000000000 |  -60    PB | ff10ffffffffffff | 0.25 PB | LDT remap for PTI
 ff11000000000000 |  -59.75 PB | ff90ffffffffffff |   32 PB | direct mapping of all physical memory (page_offset_base)
 ff91000000000000 |  -27.75 PB | ff9fffffffffffff | 3.75 PB | ... unused hole
 ffa0000000000000 |  -24    PB | ffd1ffffffffffff | 12.5 PB | vmalloc/ioremap space (vmalloc_base)
 ffd2000000000000 |  -11.5  PB | ffd3ffffffffffff |  0.5 PB | ... unused hole
 ffd4000000000000 |  -11    PB | ffd5ffffffffffff |  0.5 PB | virtual memory map (vmemmap_base)
 ffd6000000000000 |  -10.5  PB | ffdeffffffffffff | 2.25 PB | ... unused hole
 ffdf000000000000 |   -8.25 PB | fffffbffffffffff |   ~8 PB | KASAN shadow memory
__________________|____________|__________________|_________|____________________________________________________________
                                                            |
                                                            | Identical layout to the 47-bit one from here on:
____________________________________________________________|____________________________________________________________
                  |            |                  |         |
 fffffc0000000000 |   -4    TB | fffffdffffffffff |    2 TB | ... unused hole
                  |            |                  |         | vaddr_end for KASLR
 fffffe0000000000 |   -2    TB | fffffe7fffffffff |  0.5 TB | cpu_entry_area mapping
 fffffe8000000000 |   -1.5  TB | fffffeffffffffff |  0.5 TB | ... unused hole
 ffffff0000000000 |   -1    TB | ffffff7fffffffff |  0.5 TB | %esp fixup stacks
 ffffff8000000000 | -512    GB | ffffffeeffffffff |  444 GB | ... unused hole
 ffffffef00000000 |  -68    GB | fffffffeffffffff |   64 GB | EFI region mapping space
 ffffffff00000000 |   -4    GB | ffffffff7fffffff |    2 GB | ... unused hole
 ffffffff80000000 |   -2    GB | ffffffff9fffffff |  512 MB | kernel text mapping, mapped to physical address 0
 ffffffff80000000 |-2048    MB |                  |         |
 ffffffffa0000000 |-1536    MB | fffffffffeffffff | 1520 MB | module mapping space
 ffffffffff000000 |  -16    MB |                  |         |
    FIXADDR_START | ~-11    MB | ffffffffff5fffff | ~0.5 MB | kernel-internal fixmap range, variable size and offset
 ffffffffff600000 |  -10    MB | ffffffffff600fff |    4 kB | legacy vsyscall ABI
 ffffffffffe00000 |   -2    MB | ffffffffffffffff |    2 MB | ... unused hole
__________________|____________|__________________|_________|___________________________________________________________

machine dependent type

this is why dangerous use default C type

unsigned long is 8 byte wide (on x64), but 32 bit wide (on x86 arch).

I/O format of integer types

The following macros are defined in inttypes.h. Each expands to a character string literal containing a conversion specifier which can be modified by a length modifier that can be used in the format argument of a formatted input/output function when converting the corresponding integer type. These macros have the general form of PRI (character string literals for the fprintf() and fwprintf() family of functions) or SCN (character string literals for the fscanf() and fwscanf() family of functions), followed by the conversion specifier, followed by a name corresponding to a similar type name in <inttypes.h>. In these names, the suffix number represents the width of the type. For example, PRIdFAST32 can be used in a format string to print the value of an integer of type int_fast32_t.

Decimal notation

example:

#define _ISOC99_SOURCE
#include <inttypes.h>      
#include <stdio.h> 
int main(void)                                             
{                                                          

int8_t i =  40; 
printf("Demonstrating the use of the following macros:\n");
printf("Using PRId8, the printed value of 40 "                  
"is  %" PRId8"\n", i);                                    
printf("Using PRIiFAST8, the printed value of 40 "               
"is  %" PRIiFAST8"\n", i);                                 
printf("Using PRIoLEAST8, the printed value of 40 "              
"is  %" PRIoLEAST8 "\n", i);                               
return 0;                                                  
}                          


Output:

Demonstrating the use of the following macros:  
Using PRId8, the printed value of 40 is  40     
Using PRIiFAST8, the printed value of 40 is  40 
Using PRIoLEAST8, the printed value of 40 is  50

types war

recently, I face a spesific issue in order choosing best types for my program. my goal is choose best need for my C datatype

there has some header exists

• stdint.h
• sys/types.h

difference from location presefective

• /usr/lib/clang/20/include/stdint.h
• /usr/include/sys/types.h

see? stdint.h is more compiler oriented, and sys/types.h is more system oriented, that right, sys/types.h is POSIX. It's fundamentally about defining types used by system calls and other OS-level interfaces.

sys/types.h

Purpose: To define data types used in system-level programming. These types are often opaque and their actual size can vary between different systems (e.g., a 32-bit vs. a 64-bit system), but they provide a portable way to interface with the OS kernel.

Origin: System V and BSD systems, later standardized by POSIX.

Use sys/types.h when you are writing code that makes POSIX-compliant system calls (open, read, fork, stat, etc.).

stdint.h

This header is part of the ISO C99 standard and later. It is focused on providing programmers with explicit control over the size of integer data types.

Purpose: To define integer types of a specific, fixed width and to provide macros for printing and scanning these types with the printf and scanf family of functions.

conlusion

Default to <stdint.h> for all your data definitions. Only use types from <sys/types.h> when you are calling an OS/POSIX function that requires them.

normal struct? use stdint.h

interacting with os APIs, such read(), return pid, etc. use <sys/types.h>

another types wars

• uint_least8_t vs uint8_t, imagine, you on very old board, where 8 bit int is not available, this uint_least8_t will be taken as 16 bit int. meanwhile, uint8_t give you exactly 8 bit wide variable.

bits data models

this section will talk about data models.

that footage, show us what __ILP32__ is, also what is ILP32?

There has some nice websites

• https://duetorun.com/blog/20211008/data-model/#concept
• https://gcc.gnu.org/onlinedocs/gccint/Effective-Target-Keywords.html#Data-type-sizes

for example, most linux system uses LP64, which long int, and pointer are 64 bit wide. the other example is ILP32, which used by arm64ilp32, but its deprecated https://gcc.gnu.org/pipermail/gcc-patches/2025-January/673207.html

LP64 and ILP32 refer to the data model used by the language. "I" is an abbreviation that represents int type, "L" represents long type, and "P" represents the pointer type. 64 and 32 refer to the size of the data types. When the ILP32 option is used, int, long and pointers are 32-bit in size. When LP64 is used, long and pointer are 64-bit in size; int remains 32-bit. The addressing mode used by LP64 is AMODE 64, and by ILP32 is AMODE 31. In the latter case, only 31 bits within the pointer are taken to form the address. For the sake of conciseness, the terms 31-bit mode and ILP32, will be used interchangeably in this document when there is no ambiguity. The same applies to 64-bit mode and LP64. see here

Common data models

check

#include <stdio.h>

int main() {
    printf("sizeof(int): %zu\n", sizeof(int));
    printf("sizeof(long): %zu\n", sizeof(long));
    printf("sizeof(void *): %zu\n", sizeof(void *));
}

intresting topics

• https://www.ibm.com/docs/en/zos/2.4.0?topic=options-lp64-ilp32#d41843e90
• LP64 also predefined, see https://gcc.gnu.org/onlinedocs/cpp/Common-Predefined-Macros.html

ELF struct datatype

recently, I had confusion about ELF datatype. let's look into elf.h

see it?

// 32 bit elf
typedef __u32	Elf32_Word;

// 64 bit elf
typedef __u32	Elf64_Word;

why word? even in assembly says

• 8 bit -> byte
• 16 bit -> word
• 32 bit -> dword
• 64 -> qword

why word (that is 16 bit), its uses u32 instead u16?

answer

the key is portability. lets look at elf64 and elf32

typedef __u32	Elf32_Word;
typedef __u32	Elf64_Word;

keep in mind that word (in ELF) is 32 bit int. so we use this convention

• half -> unsigned 16 bit
• word -> unsigned 32 bit
• Xword -> unsigned 64 bit

also works for

• Shalf -> signed 16 bit
• Sword -> signed 32 bit
• Sxword -> signed 64 bit

finding .shstrtab section

first, we need to look at some struct in order to find shstrtab on ELF file

this is ELF header

typedef struct
{
  unsigned char	e_ident[EI_NIDENT];	/* Magic number and other info */
  Elf64_Half	e_type;			/* Object file type */
  Elf64_Half	e_machine;		/* Architecture */
  Elf64_Word	e_version;		/* Object file version */
  Elf64_Addr	e_entry;		/* Entry point virtual address */
  Elf64_Off	e_phoff;		/* Program header table file offset */
  Elf64_Off	e_shoff;		/* Section header table file offset */
  Elf64_Word	e_flags;		/* Processor-specific flags */
  Elf64_Half	e_ehsize;		/* ELF header size in bytes */
  Elf64_Half	e_phentsize;		/* Program header table entry size */
  Elf64_Half	e_phnum;		/* Program header table entry count */
  Elf64_Half	e_shentsize;		/* Section header table entry size */
  Elf64_Half	e_shnum;		/* Section header table entry count */
  Elf64_Half	e_shstrndx;		/* Section header string table index */
} Elf64_Ehdr;

this is section header

typedef struct
{
  Elf64_Word	sh_name;		/* Section name (string tbl index) */
  Elf64_Word	sh_type;		/* Section type */
  Elf64_Xword	sh_flags;		/* Section flags */
  Elf64_Addr	sh_addr;		/* Section virtual addr at execution */
  Elf64_Off	sh_offset;		/* Section file offset */
  Elf64_Xword	sh_size;		/* Section size in bytes */
  Elf64_Word	sh_link;		/* Link to another section */
  Elf64_Word	sh_info;		/* Additional section information */
  Elf64_Xword	sh_addralign;		/* Section alignment */
  Elf64_Xword	sh_entsize;		/* Entry size if section holds table */
} Elf64_Shdr;

finding it

first, we need obtain how much we need to walk, to enter the start of section header, this is can be done by

keep your mind in "Start of section headers"

access the array

because this is just start, so its started on index 0, luckly the ELF header tell me, what index .shstrtab is located, look at "Section header string table index". its 36

in C you may doing

Elf64_Shdr data = /* any data */;
dump(data[36]);

but in binary level, which what you should do is calculating how far away we need to walk from start to index of 36, which can be done by 36 * sizeof(Elf64_Shdr), in this case, this is same as 36 * ehdr->e_shentsize, then add it with initial offset where Shdr start off.

final formula ehdr->e_shoff + (shstrtab_idx * ehdr->e_shentsize)

tests

this is how shdr at index 36 look like (at binary view)

lets map it into C code for better readibility

we know that, in current section header array, .shstrtab located on 67577 far away from 0, that is location of actual .shstrtab, the Shdr only create a "list of section table, what section in this binary".

also, 374 is the size, how many memory you need to spend for this a bunch of string.

lets jump to 67577

xxd stuff

found!

future

you just need to map Elf64_Shdr on section sh_name which this data. sh_name is a offset how much we walk from 67577 until we found \0

print binary using shift and masking

8 bit int

lets play with small number first, imagine we want print 200 into binary, which is 0b11001000, this is how I do

step 0

identify how many bits we need? say, char has 1 byte, int has 4 byte, long long has 8 byte. just multiply it with 8. you'll get the bit.

• 1 * 8 = 8 bit
• 4 * 8 = 32 bit
• 8 * 8 = 64 bit

step 1

mask, lets see this pattern

0b11001000
0b10000000
---------- &
0b10000000

see? 0b10000000, which is 0x80, what about 0x8? its 0x1000. find that? by shift left the bit, we turn 0x8 into 0x80

lets follow this pattern

• 0x8 << 0 = 0x8 (4 bit)
• 0x8 << 4 = 0x80 (8 bit) <--- this is mask
• 0x8 << 8 = 0x800 (12 bit)
• 0x8 << 12 = 0x8000 (16 bit) <--- this is mask
• 0x8 << 16 = 0x80000 (20 bit)

when I want mask 8 bit int? the mask is 0x80 because its perfectly fit 0b10000000, if I want mask 16 bit int? the mask is 0x8000, because its fit 0b1000000000000000 16 bit int, can filter the MSB

conlusion: the mask that we will use, is

0x8 << BIT_SIZE - 4

SAY, BIT_SIZE is 8 (eight bit int), 8 - 4 is 4, 0x8 << 4 = 0x80 same thing with

BIT_SIZE is 16, then 16 - 4 = 12, 0x8 << 12 = 0x8000

okay, the mask has been clearly explained

step 3, shifting

say, I have 8 bit int. 0b11100011, when I mask it

n = 1

0b11100011
0b10000000
---------- &
0b10000000

lets continue shift to left

n = 2

0b11000110
0b10000000
---------- &
0b10000000

n = 3

0b10001100
0b10000000
---------- &
0b10000000

n = 4

0b00011000
0b10000000
---------- &
0b0000000

n = 5

0b00110000
0b10000000
---------- &
0b00000000

n = 6

0b01100000
0b10000000
---------- &
0b00000000

n = 7

0b11000000
0b10000000
---------- &
0b10000000

n = 8

0b10000000
0b10000000
---------- &
0b10000000

obtaining result

lets see this pattern

0b10000000 0b10000000 0b10000000 0b0000000 0b00000000 0b00000000 0b10000000 0b10000000

if we use if else, to match whatever one byte is same as 0b10000000 or not, we will get result 1110 0011

YES, we successfully turn int 8 bit into binary presentation

example

#include <stddef.h>
#include <stdio.h>

int main() {
        long long x = 3;

        size_t BIT_SIZE = sizeof(x) * 8;
        printf("size: %zu bit\n", BIT_SIZE);

        long long big_mask = (0x8ULL << (BIT_SIZE - 4));

        int counter = 0;
        for (int i = 0; i < BIT_SIZE; i++) {
                if (i % 4 == 0 && i != 0) {
                        printf("  ");
                }

                if (i % 16 == 0 && i != 0) {
                        printf("\n");
                }

                
                long int y = x & big_mask;
                printf("%d ", y == big_mask ? 1 : 0);
                x = x << 1;
        }
       
}

all about ASan & debug symbol

prerequisites

if you use clang infrastructure

• llvm-symbolizer: without this, asan offset at debug output will be binary offset, not actual source code. I will demonstrate it
• clang

if you use GCC

• addr2line (usually preinstalled)
• gcc

the tools

• gdb (optional)
• address sanitizer

testing

very simple memory buggy program

#include <stdlib.h>
#include <stdio.h>

int main() {
        char* data = malloc(1024);
        printf("%s", data);
        return 0;
}

gcc ./leak.c -o leak -g -fsanitize=address

here buggy result

lets compile it with clang clang ./leak.c -o leak -g -fsanitize=address

after llvm-symbolizer installed

asm volatile

amd64 endianness

I'll show you, the little endian (that used in most x86-64 linux systems), let go deeper

I have very simple program, the program

#include <stdio.h>

typedef unsigned char byte_ptr;

void show_bytes(byte_ptr *data, size_t n) {
        int i = 0;
        for(; i < n; i++) {
                printf("%p -> %.2x\n",  &data[i], data[i]);
        }

        printf("\n");
}

int main() {
        unsigned long da = 0x5894F93;
        show_bytes((unsigned char*)&da, 8);
}

as you can see, this is what we write

hexadecimal	: 0x05 0x89 0x4F 0x93
dummy addr  : 0    1    2    3

the result

0x7fff0fe80d10 -> 93
0x7fff0fe80d11 -> 4f
0x7fff0fe80d12 -> 89
0x7fff0fe80d13 -> 05
0x7fff0fe80d14 -> 00
0x7fff0fe80d15 -> 00
0x7fff0fe80d16 -> 00
0x7fff0fe80d17 -> 00

wow, it reversed!, because

hexcode      : 0x93 0x4f 0x89 0x05 0x00 0x00 0x00 0x00
dummy addr   : 0    1    2    3    4    5    6    7
               ^~~ MSB                            ^~~ LSB

(read it first from right, to left, as you read binary code) WOW, its least endian first, so the machine is little-endian

you can also check out my lscpu result

rust

result

big endian

because most of machine is little-endian, even RP2040 ARM chip, I'll demontrate it (with some stuff)

first, I'll use miri

miri

this unique tool can emulate big endian

Rust stuff

Welcome in this rust stuff section.

Rust cursor, How it works

Cursor in rust basically just a in-memory implementation of Seek. see fseek() here

Imagine, we have a file. and this is representation of contents

abcde

seek start from 1, is bcde, start from 2 is cde and so-on, this is also happen when we use seek from end, for example, seek end -1 is mean e, -2 is de, etc.

From code perspective

this is signature of cursor

pub struct Cursor<T> {
    inner: T,
    pos: u64,
}

and also, this struct implement Write and Seek

so, in order to understanding this concept, let code first in C

#include <string.h>
#include <stdlib.h>
#include <stdio.h>

struct seekfd {
    FILE                *file;
};

int start_open(struct seekfd *seekfd) {
    seekfd->file = fopen("a.txt", "r");

    if (seekfd->file == NULL) {
        fprintf(stderr, "%s", "error");
        return -1;
    }
}

int do_op(struct seekfd *seekfd) {
    char allocated_buf[2048];
    memset(allocated_buf, 0, 2048);

    fseek(seekfd->file, 1, SEEK_SET);
    int ret = fread(allocated_buf, 1, 3, seekfd->file);

    printf("%s", allocated_buf);
}

int main() {
    struct seekfd seekfd;

    int ret = start_open(&seekfd);
    if (ret < 0) {
        return -9;
    }

    ret = do_op(&seekfd);
    if (ret < 0) {
        return ret;
    }
}

compile with

clang fseek.c -o p -g

next step, create dummy files named a.txt with content abcdefghijklmnopqr

first run with fseek(seekfd->file, 0, SEEK_SET); = abc second run with fseek(seekfd->file, 1, SEEK_SET); = bcd etc...

this concept is basically same in Rust

Rust context

this is small example show Seek works (in-memory) instead utilize real file

use std::io::prelude::*;
use std::io::{self, SeekFrom};
use std::io::Cursor;

fn write_bytes<W: Write + Seek>(mut w: W) -> io::Result<()> {
    w.seek(SeekFrom::Start(5))?;
    
    for i in 1..6 {
        w.write(&[i])?;
    }

    Ok(())
}


fn main() {
    let mut buf = Cursor::new(vec![0; 20]);
    write_bytes(&mut buf).unwrap();
    println!("{:?}", &buf.get_ref()); // print vector
}

this is the output

[0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

instead, we change SeekFrom::Start(5) with SeekFrom::Start(10), the write will started at offset 10. this is the output

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 0, 0, 0, 0, 0]

Rust bytes has_remaining() method

this function return whatever any best left on buffer. example, let make simple tcp/ip

impl Connection {
    pub fn new(stream: TcpStream) -> Connection {
        Connection {
            stream: BufWriter::new(stream),
            buffer: BytesMut::with_capacity(4096)
        }

    }

    pub async fn read(&mut self) 
        -> std::result::Result<Option<Frame>, Box<dyn std::error::Error + Send + Sync>> {
        
        let _ = self.stream.read_buf(&mut self.buffer).await;
        println!("hasremaining: {}", self.buffer.has_remaining());
        
        while self.buffer.has_remaining() {
            let bytes = self.buffer.get_u8();
            println!("{} -> {}", bytes as char, self.buffer.has_remaining());
        }

        println!("{}", self.buffer.has_remaining());
        // Frame::check(&mut self.buffer);

        Ok(Some(Frame::Null))
    }
}

let's see the server result using ncat

which the input is something look like this

[22:55:41] fadhil_riyanto@integral2 /home/fadhil_riyanto [SIGINT]
> ncat 127.0.0.1 6380
abc

from this output, we know

hasremaining: true <--- this is initial, didn't mean anything
a -> true <--- a already read, but next char is b, why this is true
b -> true <--- next is c, also true
c -> true <--- next is \n char, also true

 -> false <--- this is false because no next chars
false <--- confirm

Also, get_u8 returns the first byte in the buffer. Each time it's called, it advances the position, so subsequent calls return the next bytes.

How Rust into() Box + Error analysis

suppose, we have function return signature like this

std::result::Result<Option<Frame>, Box<dyn std::error::Error + Send + Sync>>

and then, we return something like this from our function

return Err("connection reset".into())

how the into() works?

analysis

look at this snippet

#[cfg(not(no_global_oom_handling))]
#[stable(feature = "rust1", since = "1.0.0")]
impl<'a> From<&str> for Box<dyn Error + Send + Sync + 'a> {
    /// Converts a [`str`] into a box of dyn [`Error`] + [`Send`] + [`Sync`].
    ///
    /// [`str`]: prim@str
    ///
    /// # Examples
    ///
    /// ```
    /// use std::error::Error;
    /// use std::mem;
    ///
    /// let a_str_error = "a str error";
    /// let a_boxed_error = Box::<dyn Error + Send + Sync>::from(a_str_error);
    /// assert!(
    ///     mem::size_of::<Box<dyn Error + Send + Sync>>() == mem::size_of_val(&a_boxed_error))
    /// ```
    #[inline]
    fn from(err: &str) -> Box<dyn Error + Send + Sync + 'a> {
        From::from(String::from(err))
    }
}

if we go deeper into From::from, we found this

#[inline]
    fn from(err: String) -> Box<dyn Error + Send + Sync + 'a> {
        struct StringError(String);

        impl Error for StringError {
            #[allow(deprecated)]
            fn description(&self) -> &str {
                &self.0
            }
        }

        impl fmt::Display for StringError {
            fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
                fmt::Display::fmt(&self.0, f)
            }
        }

        // Purposefully skip printing "StringError(..)"
        impl fmt::Debug for StringError {
            fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
                fmt::Debug::fmt(&self.0, f)
            }
        }

        Box::new(StringError(err))
    }

nb: actually StringError is string aliases. then, we see in this section. we created a heap-allocated using Box::new(StringError(err)).

Rust buffer advance

this simple article show how buffer advance works

use bytes::Buf;

fn main() {
    let mut buf = &b"hello world"[..];

    println!("{}", String::from_utf8_lossy(buf.chunk()));

    buf.advance(6);
    println!("{}", String::from_utf8_lossy(buf.chunk()));
}

and the result

hello world
world

Rust RESP get_line analysis

if we look at simple tokio redis example (mini redis), we can see how the data is parsed. look at this

the resp is generally read something like this `+abc\r\n'

this is the code that actually crop down the buffer

fn get_line<'a>(src: &mut Cursor<&'a [u8]>) -> Result<&'a [u8], Error> {
    // Scan the bytes directly
    let start = src.position() as usize;
    // Scan to the second to last byte
    let end = src.get_ref().len() - 1;

    for i in start..end {
        if src.get_ref()[i] == b'\r' && src.get_ref()[i + 1] == b'\n' {
            // We found a line, update the position to be *after* the \n
            src.set_position((i + 2) as u64);

            // Return the line
            return Ok(&src.get_ref()[start..i]);
        }
    }

    Err(Error::Incomplete)
}

and I will show you how they works (algorithm)

variables setup

we have start and end, which consist of

let start = src.position() as usize;
let end = src.get_ref().len() - 1;

the src.position() always return 0, also depending on the context. src.get_ref().len() return the length of the buffer, for example, +rand\n\n has length 7.

because we wan't to read two chars at the end, which is \r and \n

so we substract length with 1, this is avoid buffer overflow

suppose this situation: buf: +rand\n\n

when we add everything with 1, we got this idx: 012345 6 7, it is idx[7]is found? no, so we need to sub with 1, the idx is something like this idx: 012345 6, no overflow

then we check the i and i + 1, make sure there is \r\n

set position

now, we set the cursor to the next incoming buffer. which after \r\n

src.set_position((i + 2) as u64);

suppose:

+rand\r\n+randomsomething\r\n
012345 6 ^ 
^^   ^   this is i + 2, indicates next buffer
ii+1 orig i

postgresql - get() generic function signature

In someday, i found

#[track_caller]
    pub fn get<'a, I, T>(&'a self, idx: I) -> T
    where
        I: RowIndex + fmt::Display,
        T: FromSql<'a>,
    {
        match self.get_inner(&idx) {
            Ok(ok) => ok,
            Err(err) => panic!("error retrieving column {}: {}", idx, err),
        }
    }

some sort of notes:

• https://doc.rust-lang.org/rust-by-example/generics.html
• https://serokell.io/blog/rust-generics

how to call it

let data2 = data_actual[0].get::<_, i32>(0);

Rust build logging

rust build usually don't show any output if you try to println!() in build.rs, in order to force this, use

println!("cargo:warning=start build!");

result

warning: ourcrate-rs@0.1.0: Something went wrong!

all docs: https://doc.rust-lang.org/cargo/reference/build-scripts.html

Rust tonic gRPC server build process

topik ini khusus membahas gRPC di rust,

build

contoh template build rs:

use std::{env, path::PathBuf};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap());
    let out_dir = PathBuf::from(
        "/home/fadhil_riyanto/BALI64/fadev-service-rs/rpc/src/pb".to_string()
    );

    println!("cargo:warning=start build!");
    
    tonic_build::configure()
        .build_server(true)
        .build_client(false)
        .file_descriptor_set_path(out_dir.join("random_service.bin"))
        .compile_protos(
            &["proto/random_service.proto"], 
            &["proto"]
        )?;


    Ok(())
}

nb:

• file_descriptor_set_path: dipakai sebagai reflection, bakal ketemu kalau pakai grpcurl.
• build_server dan build_client: disini ada 2 mode build, untuk server atau untuk client. maka kita set server true, client false
• compile_protos: masing2 berisi 2 param yg menerima slice, param pertama berisi lokasi protobuf, param kedua berisi root foldernya.

anda bisa lihat-lihat contoh lain disini: https://github.com/hyperium/tonic/tree/master/examples

optional:

• use .out_dir() ketimbang harus nurut dengan apa yang OUT_DIR katakan

OUT_DIR

what it is, simplenya OUT_DIR ini akan diset jika protobuf sudah di compile, dalam artian, file *.proto diubah jadi kumpulan file *.rs yang bakal di include sama macro, kemudian di compile

urutan compile adalah, macro dievaluasi dulu, baru compile, sampai sini masuk akal. ini macro signature nya

#[macro_export]
macro_rules! include_proto {
    ($package: tt) => {
        include!(concat!(env!("OUT_DIR"), concat!("/", $package, ".rs")));
    };
}

tambahan info bisa baca2 sendiri disini: https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-build-scripts, dan jg biasanya ini akan diinclude ke sebuah file handler. kira kira seperti itu

Rust tokio-postgres connection config example

first, this is example from the documentation

 let (client, conn_ctx) = tokio_postgres::connect("host=localhost user=postgres", NoTls);

then, we want to convert it into config like ctx. see this

use tokio_postgres::{NoTls, Error};

#[tokio::main]
async fn main() {
        let mut config = postgres::config::Config::new();
        config.host("localhost");
        config.user("fadhil_riyanto");
        config.password("1921684338");

        let client_ret = config.connect(NoTls);


}

miri rust setup

Embedtronics

picotool build

os setup

sudo pacman -S cmake arm-none-eabi-gcc arm-none-eabi-newlib

setup picosdk

git clone https://github.com/raspberrypi/pico-sdk.git
cd pico-sdk
git submodule update --init
export PICO_SDK_PATH=$PWD

build picotool

git clone https://github.com/raspberrypi/picotool.git
cd picotool
mkdir build
cd build
cmake ..
make

if you get cmake error minimum etc..., use cmake .. -DCMAKE_POLICY_VERSION_MINIMUM=3.5 instead

Math

Welcome in this section.

Set operations formula

Union of Two Sets (cardinality)

\[ |A \cup B| = |A| + |B| - |A \cap B| \]

how to read: whole A and B add together, its make a data duplication on center of venn diagram, we remove that duplication using \( |A \cap B| \)

Intersection (steps)

its flip of Union \[ |A \cup B| = |A| + |B| - |A \cap B| \] \[ |A \cap B| = |A| + |B| - |A \cup B| \]

Intersection of three sets

\[ |A \cup B \cup C| = |A| + |B| + |C| - |A \cap B| - |A \cap C| - |B \cap C| + |A \cap B \cap C| \]

how it works, de remove data duplication two times on \( |A \cap B| \), \( |B \cap C| \), and \( |B \cap C| \), then fill the empty section on the center which \( |A \cap B \cap C| \)

This is proof of concept by nice guy on internet: https://www.youtube.com/watch?v=vVZwe3TCJT8.

Difference

\[ |A - B| = |A| - |A \cup B| \]

Symetric difference

1. General formula

\[ A \space \triangle \space B = (A \cup B) - (A \cap B) \] Cardinals version: \[ | A \space \triangle \space B | = |A - B| + |B - A| \] \[ | A \space \triangle \space B | = |A| + |B| - 2|A \cap B| \]

2. Symetric difference properties

• \( A \space \triangle \space B = B \space \triangle \space A \)

• \( (A \space \triangle \space B) \space \triangle C = A \space \triangle \space (B \space \triangle C) \)

• \( (A \space \triangle \space \emptyset) = A \), why?

  • \(A \space \triangle \space B = (A \cup B) - (B \cap A) \)
  • \(A \space \triangle \space \emptyset = (A \cup \emptyset) - (\emptyset \cap A) \), but
  • \( (A \cup \emptyset) = A \) and \( (\emptyset \cap A) = \emptyset \)
  • \( (A - \emptyset) = A \)

• \( (A \space \triangle \space A) = \emptyset \)

Set stuff

there is some intresting question about discrete math

why \( \mathcal{P}(\emptyset) = \lbrace \emptyset \rbrace \)

consider \( \mathcal(S) \) where \( S \) is a set, for example

• if \( S = \lbrace a \rbrace \), then \( \mathcal{P}(S) = \lbrace \emptyset, \lbrace a \rbrace \rbrace \), same as \( \mathcal{P}(a) \)
• if \( S = \lbrace a, b \rbrace \), then \( \mathcal{P}(S) = \lbrace \emptyset, \lbrace a \rbrace, \lbrace b \rbrace, \lbrace a, b \rbrace \rbrace \), same as \( \mathcal{P}(a, b) \)
• if \( S = \lbrace \emptyset \rbrace \), then \( \mathcal{P}(S) = \lbrace \emptyset, \lbrace \emptyset \rbrace \rbrace \), same as \( \mathcal{P}(\emptyset) \)

so, because \( \lbrace \rbrace \) usually hidden, \( \mathcal{P}(\emptyset) = \lbrace \emptyset \rbrace \)

let \( A = \emptyset, B = \emptyset \), how to \( A \times B \)

Consider \( A \times B = \lbrace (a,b) | a \in A, b \in B \rbrace \)

because

• A is empty, no elements can be paired
• B is also empty

conlusion: no paired element = empty

proof by cardinality \( |A| = 0, |B| = 0, \text{so, when } |A \times B| = |A| \times |B| = 0 \times 0 = 0 \)

does \( P(\emptyset) = \lbrace \lbrace \rbrace \rbrace = \lbrace \emptyset \rbrace \) ?

true, this is due \( \lbrace \emptyset \rbrace \) same as \( \lbrace \lbrace \rbrace \rbrace \), then consider example \( P(a) = \lbrace \lbrace \rbrace, \lbrace a \rbrace\rbrace \)

Suppose X and Z are independent of each other

• \( | \mathcal{P}(X \cap Z)| \)
• \( | X - Z | \)
• \( | X \oplus Z | \)
• \( | X \cap Z | \)
• \( | \mathcal{P}(X) \cup \mathcal{P}(Z)| \)
• \( | \overline{\mathcal{P}(X) \cup \mathcal{P}(Z)}| \)

nb: bagian ini belum jadi

Logika

Ringkasan ini dibuat sebagian sebagian dari Kuliah IF1220 Matematika Diskrit ITB oleh bapak Rinaldi Munir. dan berbagai sumber seperti Wikipedia, ChatGPT dsb

Proposisi

adalah pernyataan bahwa suatu dua perbandingan adalah sama besar, contoh

• \( \frac{a}{b}=\frac{c}{d} \)
• \( 4 = 2 * 2 \)
• 13 adalah bilangan ganjil
• Untuk sembarang bilangan bulat \( n >= 0 \), maka \( 2n \) adalah genap

contoh yang bukan

• Isilah gelas tersebut dengan air
• \( x + 3 = 8 \)
• \( x > 3 \)

dan, Proposisi dilambangkan dengan huruf kecil seperti \( p, q, r, ... \)

Bentuk bentuk nya

• Proposisi atomik bentuk proposisi tunggal, contoh
  • 2n selalu genap untuk n=0, 1, 2, …
  • Ibukota Maluku Utara adalah Ternate
• proposisi majemuk diantaranya:
  • conjunction (and): \( \wedge \)
  • disjunction (or): \( \vee \)
  • negation (!n): \( ~ \)
  • exclusive disjunction (xor): \( \oplus \)

Table

nb:

• Tautologi adalah pernyataan logika yang selalu benar, tidak peduli nilai kebenaran dari komponen-komponennya. contoh

• Kontradiksi adalah pernyataan logika yang selalu salah, tidak peduli nilai kebenaran dari komponen-komponennya. contoh

Hukum hukum nya

Implikasi

Disebut juga proposisi bersyarat, seperti jika x maka y, notasinya \( p \rightarrow q\). \( p \) nya adalah condition, \( q \) nya adalah conlusion

versi versinya jika dijadikan teks

• Jika p, maka q (if p, then q)
• Jika p, q (if p, q)
• p mengakibatkan q (p implies q)
• q jika p (q if p)
• p hanya jika q (p only if q)
• p syarat cukup untuk q (p is sufficient condition for q)
• q syarat perlu bagi p (q is necessary condition for q)
• q bilamana p (q whenever p)
• q mengikuti dari p (q follows from p)

penjelasan kenapa \( F \rightarrow F = T \)

“If I win the lottery, I’ll buy you a car.”
I didn't win the lottery → (False)

I didn't buy a car → (False)

dan juga \( P \rightarrow Q \) sebenarnya sama dengan \( \sim{P} \vee Q \)

Penjelasan kenapa \( \sim (p \rightarrow q )\) itu sama dengan \( p \space \wedge \sim{q} \)

Kita tahu bahwa \( p \rightarrow q \) itu sebenarnya sama dengan \( \sim{p} \vee p \) , maka steps yang dibutuhkan hanya

• \( \sim(p \rightarrow q) \)
• \( \sim(\sim{p} \vee q) \)
• \( p \space \wedge \sim{q} \)

tabel varian implikasi (Proporsi bersyarat)

• Konvers (kebalikan): \( q \rightarrow p \)
• Invers : \( \sim p \rightarrow \sim q \)
• Kontraposisi : \( \sim q \rightarrow \sim p \)

Bi-impication

Intinya, operand kanan kiri harus sama, entah sama sama true, atau sama sama false. notasinya: \( p \leftrightarrow q \)

tabel kebenaran

contoh

analogi simple:

• Jika suatu bilangan genap, maka habis dibagi 2: \( \text{true} \leftrightarrow \text{true} = \text{true} \)

• Jika suatu bilangan bukan genap, maka tidak akan habis dibagi 2: \( \text{false} \leftrightarrow \text{false} = \text{true} \)

kumpulan soal Matematika Diskrit

No. 1

Dalam sebuah survei terhadap 120 orang, diperoleh data sebagai berikut:

• 65 orang suka kopi
• 70 orang suka teh
• 50 orang suka cokelat
• 30 orang suka kopi dan teh
• 28 orang suka kopi dan cokelat
• 26 orang suka teh dan cokelat
• 15 orang suka semua minuman
• Semua orang minimal menyukai satu jenis minuman

Tentukan:

1. Banyak orang yang hanya suka kopi
2. Banyak orang yang suka teh dan cokelat, tapi tidak suka kopi
3. Banyak orang yang suka tepat dua jenis minuman
4. Banyak orang yang hanya suka satu jenis minuman

No. 2

Jika diketahui

• S = {1,2,3 …, 15},
• A = { 2,3,5,7},
• B = { 1,3,5,7,9},
• C = {2,4,6,8}

Tentukan :

1. A ∪ B ∪ C
2. A ∩ C
3. B - A
4. (A’ ∩ B) - C
5. (B - C) ∩ A’

No. 3

Dalam survei terhadap 80 mahasiswa, diperoleh data:

• 45 mahasiswa suka Matematika
• 38 mahasiswa suka Fisika
• 40 mahasiswa suka Kimia
• 20 mahasiswa suka Matematika dan Fisika
• 25 mahasiswa suka Fisika dan Kimia
• 18 mahasiswa suka Matematika dan Kimia
• 10 mahasiswa suka ketiga mata kuliah
• 8 mahasiswa tidak suka semua mata kuliah

tentukan

1. Banyak mahasiswa yang suka hanya Matematika
2. Banyak mahasiswa yang suka tepat dua mata kuliah
3. Banyak mahasiswa yang suka Matematika atau Kimia tapi tidak Fisika
4. Banyak mahasiswa yang suka hanya satu mata kuliah

format latex menyusul

asal usul rumus sudut vektor

awal mula dari rumus entah berantah ini. link untuk oprek rumus: https://proofwiki.org/wiki/Cosine_Formula_for_Dot_Product

\[ \vec{A} \cdot \vec{B} = |\vec{A}| |\vec{B}| cos(\theta) \]

lalu \[ \frac{\vec{A} \cdot \vec{B}}{|\vec{A}| |\vec{B}|} = cos(\theta) \]

lalu kita bisa balik, menjadi \[ cos(\theta) = \frac{\vec{A} \cdot \vec{B}}{|\vec{A}| |\vec{B}|} \]

\[ arccos(\frac{\vec{A} \cdot \vec{B}}{|\vec{A}| |\vec{B}|}) = \theta \]

solved!

transformasi elementer

beberapa notasi aneh disini diderive dari PPT kampus.

• \( H_{ij}A \): Swap baris i dengan baris j
• \( H_i^{(k)} \): mengalikan baris i dengan konstanta k
• \( H_{ij}^{(k)} \): hitung dulu konstanta k dikali baris j, lalu hasilnya ditambah baris i, simpan jg hasilnya ke baris i
• \( H_{i \space \space \space \space \space \space \space j}^{\space \space (k) \space \space \space (z)}(A)_i \): intinya, baris ke i dikalikan dgn k, kalau udah lanjut hitung baris j dikalikan z. setelah itu hasilnya tentusaja simpan ke i hasil itung2 an tadi.

matrix invers

for 2x2

\[ A^{-1} = \frac{1}{(a \cdot d) - (b \cdot c)} \begin{bmatrix} d & -b & \newline -c & a \end{bmatrix}\]

for 3x3 \[ A^{-1} = \frac{1}{\det A} \text{adj } A \]

Cryptography

Alur konek ke RPC server telegram

step ini dibuat oleh seseorang dari masa lalu sekitar tahun 2020 yang akunnya telah terhapus, credit kepada Butthx

`originals written in telegram chat, but because I want to preserve it. I write it here

step 1:

Bikin koneksi ke server telegram pakai TCP yang tersedia (disini gwa pakai TCPIntermediate)

step 2:

Bikin random nonce dengan value int128

step 3:

Kirim bytes dari ReqPqMulti dengan value bytes dari nonce tadi.

step 4:

pick salah satu dari beberapa publick fingerprints yang diberikan dari results step3.

step 5:

Lakuin faktor bilangan pq yang diberikan dari results step3, dapetin lagi hasil pembagian dari pq tadi dengan hasil faktor, terus urutkan dari yang terkecil. untuk yang kecil di kasih nama p dan yang besar kasih nama g.

step 6:

bikin nonce baru (jangan ubah nonce yang lama) dengan value random int256.

step 7:

bikin hash (sha1) dari bytes PQInnerData ke telegram: bytes[pq,bytes p,bytes q,nonce,nonce baru,server nonce (dapet dari step3)]

step 8:

bikin padding dengan random bytes dengan panjang -(length bytes PQInnerData + length sha1) mod 255

step 9:

encrypt bytes[sha,bytes PQInnerData,padding] dengan RSA (pakai publik fingerprints tadi)

step 10:

kirim bytes ReqDhParams: bytes[nonce,server nonce, bytes p, bytes q, encrypted data (step9) ,fingerprint]

step 11:

bikin temporary aes key: bytes[sha1[bytes nonce baru, bytes server nonce],sha1[bytes server nonce, bytes nonce baru] slice 0 – 12]

step 12:

bikin temporary aes initial vector (IV): bytes[sha1[bytes server nonce, bytes nonce baru] slice 12 – end, sha1[bytes nonce baru,bytes nonce baru], bytes[nonce baru] slice 0 – 4 ]

step 13:

decrypt encryptedAnswer (dapet dari step 10) menggunakan AES-256-IGE pakai key sama iv tadi.

step 14:

parse decrypted answer tadi ke TLSchema/TLObject

step 15:

bikin random int(panjang 256 bytes, bukan int256 ygy) kita beri nama b

step 16:

dapetin g_b dengan (g ** b) mod dh_prime

• g, dh_prime → dapet dari step 14

step 17:

bikin hash sha1 dari bytes[ClientDhInnerData[nonce, server nonce, 0,unsigned big bytes g_b length 256]]

step 18:

bikin padding dengan length -(length bytes ClientDhInnerData + length sha1) mod 16

step 19:

encrypt bytes[bytes step 17,bytes ClientDhInnerData, padding ] dengan AES-256-IGE, key sama iv masih menggunakan yang step 11.

step 20:

kirim bytes[SetClientDhParams[nonce, server nonce, bytes step 19]]

step 21:

Bikin auth key dengan big-unsigned-bytes[big-u-int(g_a) ** b mod dhPrime] length 256

step 22:

Lakuin security check..

notes: Untuk format pengiriman bytes:

bytes[sessionId,messageId,length content,content]
karena belum ada authkey, jadi sessionId  = Buffer.alloc(8)

untuk dapetin real data dari telegram, slice bytes yang di terima dari 20 – end

pengiriman content selalu bentuk bytes, jadi semua parameter jadiin bytes :)

/**
 * tgsnake - Telegram MTProto framework for nodejs.
 * Copyright (C) 2022 butthx <https://github.com/butthx>
 *
 * THIS FILE IS PART OF TGSNAKE
 *
 * tgsnake is a free software : you can redistribute it and/or modify
 * it under the terms of the MIT License as published.
 */

export const DCTest = {
  1: '149.154.175.10',
  2: '149.154.167.40',
  3: '149.154.175.117',
};
export const DCProd = {
  1: '149.154.175.53',
  2: '149.154.167.51',
  3: '149.154.175.100',
  4: '149.154.167.91',
  5: '91.108.56.130',
};
export const DCProdMedia = {
  2: '149.154.167.151',
  4: '149.154.164.250',
};
export const DCTestIPV6 = {
  1: '2001:b28:f23d:f001::e',
  2: '2001:67c:4e8:f002::e',
  3: '2001:b28:f23d:f003::e',
};
export const DCProdIPV6 = {
  1: '2001:b28:f23d:f001::a',
  2: '2001:67c:4e8:f002::a',
  3: '2001:b28:f23d:f003::a',
  4: '2001:67c:4e8:f004::a',
  5: '2001:b28:f23f:f005::a',
};
export const DCProdMediaIPV6 = {
  2: '2001:067c:04e8:f002:0000:0000:0000:000b',
  4: '2001:067c:04e8:f004:0000:0000:0000:000b',
};
export function DataCenter(
  dcId: number,
  testMode: boolean,
  ipv6: boolean,
  media: boolean
): [ip: string, port: number] {
  if (testMode) {
    return [ipv6 ? DCTestIPV6[dcId] : DCTest[dcId], 80];
  } else {
    if (media) {
      return [ipv6 ? DCProdMediaIPV6[dcId] : DCProdMedia[dcId], 443];
    } else {
      return [ipv6 ? DCProdIPV6[dcId] : DCProd[dcId], 443];
    }
  }
}

ca-certificates

draft: mengoprek kriptografi dibalik HTTPS & konsep public private key pair dan juga peran CA authority, kenapa mereka harus dimasukkan ke package ???

dan juga kenapa MITM (Man in the Middle) bisa melogging jaringan, ketika certificate telah dipasang. kenapa hanya dengan certificate mitm bisa mendecode data yang lewat.

PAP, CHAP, EAP

Server

Cgit setup

Ku mau share pengalaman bikin self hosting git server, pakai cgit, biar engga kalah aja sama git.kernel.org dan kernel.dk wkwk

step satu, include dir nya sites-available spt biasa, contoh config nya https://gist.github.com/fadhil-riyanto/328f2cc0607eb58b713cdeaa473d08ee, lalu config untuk sites-available nya https://gist.github.com/fadhil-riyanto/c2e26de30c97600e4ee9342aefbd6ece

nb: install fcgiwrap dulu

lalu edit /etc/cgitrc, contoh config: https://gist.github.com/fadhil-riyanto/4033e6ce59a35568c31dbd5e5a1d29bd

harusnya sampai sini, cgit dah bisa diakses, cuman empty repository aja.

ok, lalu kita ke permission2 nya, pertama2 pakai akun root, tldr

cd /root
mkdir git
cd git
mkdir user1, misal aja mkdir fadhil_riyanto
cd fadhil_riyanto

clone git apapun disini, misal git clone https://github.com/fadhil-riyanto/reponame.git --bare (harus pakai bare)

posisi dir ini di /root/git/fadhil-riyanto/reponame.git

ok, balik ke parent dir, posisi di /root/git lakukan chown -R fadhil_riyanto fadhil-riyanto, jika engga nanti error "git submodule update" failed with 'fatal: detected dubious ownership in repository at...' etc...

ok, saatnya push git remote add cgit fadhil_riyanto@123.123.111.222:/root/git/fadhil-riyanto/reponame.git

config HTTPS nginx

Cara config HTTPS di server pakai certbot (NGINX)

• apt install certbot python3-certbot-nginx
• verify config na pakai nginx -t
• generate cert na pakai certbot --nginx -d namasite.com

ganti namasite.com dengan embel2 di /etc/nginx/sites-enabled/somefile.conf

didalamnya kan ada server_name, nah diganti dgn itu

Setup virtual network interface

cara setup virtual ethernet + assign ip nya

• sudo ip link add veth0 type dummy

bisa dicek pakai ip addr

[11:11:35] fadhil_riyanto@integral2 /etc/systemd/network  
> ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 2c:4d:54:c6:f2:0c brd ff:ff:ff:ff:ff:ff
    altname enx2c4d54c6f20c
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f0:03:8c:66:9c:21 brd ff:ff:ff:ff:ff:ff
    inet 192.168.5.44/24 brd 192.168.5.255 scope global dynamic noprefixroute wlan0
       valid_lft 581sec preferred_lft 581sec
    inet 192.168.5.197/24 metric 1024 brd 192.168.5.255 scope global secondary dynamic wlan0
       valid_lft 582sec preferred_lft 582sec
    inet6 fe80::5574:5771:be47:2f7b/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::f203:8cff:fe66:9c21/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: veth0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 76:4d:3f:3d:11:7f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::744d:3fff:fe3d:117f/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

lalu assign ip nya

• sudo ip addr add 192.168.100.10/24 dev veth0
• sudo ip link set veth0 up

dan bisa diping juga

untuk remove interface nya bisa pakai sudo ip link delete vnet0

some sorts of equalivalent command

• Create bridge: brctl addbr br0 ip link add br0 type bridge
• Delete bridge: brctl delbr br0 ip link delete br0
• Add interface to bridge: brctl addif br0 eth0 ip link set eth0 master br0
• Remove interface from bridge: brctl delif br0 eth0 ip link set eth0 nomaster
• Show bridge info: brctl show or brctl show br0 bridge link show or ip link show master br0
• Enable/disable STP: brctl stp br0 on/off bridge link set dev eth0 guard off/on (note: STP enable/disable logic is reversed)

ip aliasing

satu interface bisa punya lebih dari 1 IP dinamai sebagai IP aliasing. contoh

eth0 bisa punya 2 ip dengan cara

• ip addr add 192.168.1.100/24 dev eth0
• ip addr add 192.168.1.101/24 dev eth0

atau bisa juga ada 2 interface, katakanlah eth0 dengan eth1, masing2 punya ip nya sendiri2 dengan cara

ip addr add 192.168.1.10/24 dev eth0
ip addr add 10.0.0.5/24 dev eth1

tambahan

untuk ngeremove salah satu ip nya, bisa dengan ip addr del <IP NYA>/<CIDR> dev <IFACE>

Setup bridge enp0s16u1

• setup dulu bridgenya pakai ip link add br0 type bridge
• buat enp0s16u1 sebagai master dari br0, pakai ip link set enp0s16u1 master br0
  • catatan, wlan0 tidak bisa di add. https://bbs.archlinux.org/viewtopic.php?id=264651, https://wiki.archlinux.org/title/Network_bridge#Wireless_interface_on_a_bridge
• lalu enable semua interface nya
  • sudo ip link set dev enp0s16u1 up
  • sudo ip link set dev br0 up
• set dhcp sudo dhclient br0
  • nb, ditahap ini, harusnya sudah dapat ip dan bisa ping keluar.
• hapus ip bawaan enp0s16u1 dan pakai br0 sebagai default interface pakai sudo ip addr flush dev enp0s16u1

bisa di test dengan

• hapus dahulu ip yang tadi diberikan ke br0 pakai sudo dhclient -r br0, pastikan ip nya kosong
• lalu hapus ip milik enp0s16u1
• test ping
• lalu lakukan lagi sudo dhclient br0

reverse command

Pakai ini untuk in-case ada kesalahan

hapus ip br0

• ip addr show br0
• sudo ip addr del 192.168.1.10/24 dev br0

tambahan, kadang tidak bener2 kehapus karna mungkin NetworkManager, systemd-networkd, dhclient aktif.

nb: untuk dhcp kadang NetworkManager sama systemd-networkd bertabrakan, solusinya stop dahulu pakai

• sudo systemctl stop systemd-networkd.socket
• sudo systemctl stop systemd-networkd

next, buat bridge ke VM. topologi

   [Internet]
       |
     Router
       |
     eth0 (on host) → br0 (bridge)
                      |
    +--------+--------+--------+
    |        |        |        |
  VM1      VM2      VM3      etc.
  ↳ 192.168.12.101   192.168.12.102   ...

mulai dari sini akan utak atik veth

setup veth manual

• sudo ip link add veth0 type veth peer name veth0-peer
• sudo ip link set veth0 master br0
• sudo ip link set veth0 up
• sudo ip link set veth0-peer up
• sudo dhclient veth0-peer

topologinya kira2 seperti ini

                +---------------------------+
                |        Linux Host         |
                |                           |
                |   +-------------------+   |
                |   |       br0         |   |  ← bridge acting as a virtual switch
                |   +--------+----------+   |
                |            |              |
                |         veth0            ... ← more veths for other containers/VMs
                |            |
                +------------+--------------+
                             |
                             | (virtual cable)
                             |
                         veth0-peer
                             |
                    +------------------+
                    |   Container/VM   |  ← gets its own IP
                    +------------------+

lanjut assign veth-peer ke qemu (pakai tap device)

topology:

            Linux Host
       +------------------+
       |      br0         |  ← acts as switch
       +--+-----------+---+
          |           |
        eth0         tap0  ← tap device for QEMU VM
                        |
                   QEMU VM

attach

sudo ip tuntap add dev tap0 mode tap
sudo ip link set tap0 up
sudo ip link set tap0 master br0

qemu

sudo qemu-system-x86_64 \
  -enable-kvm \
  -m 2048 \
  -hda ubuntu.qcow2 \
  -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
  -device virtio-net-pci,netdev=net0

test qemu mikrotik

qemu-system-x86_64 \
                -enable-kvm \
                -boot order=d \
                -drive file=chr.qcow2,format=qcow2 \
                -m 4G \
                -smp 4 \
                -net user,hostfwd=tcp::10022-:22,hostfwd=tcp::8080-:8080 \
                -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
                -device virtio-net-pci,netdev=net0

untuk test nya, bisa dengan

• cek ip mikrotik pakai /ip address print
• lalu ip netns exec namespace_satu ping <IP NYA>
• Bisa juga pakai dummy http server php, caranya sudo ip netns exec namespace_satu php -S 192.168.12.82:1234 (cek dahulu ip namespace pakai ip addr)

contoh setup pakai namespace

• create dulu pakai sudo ip netns add namespace_satu
• set sudo ip link set veth0-peer netns namespace_satu
• cek dahulu, pastikan didalamnnya masih kosong ip nya, pakai sudo ip netns exec namespace_satu ip addr
• lalu kita nyalakan veth0-peer nya sudo ip netns exec namespace_satu ip link set dev veth0-peer up, tapi tetap saja masih belum dapat ip. maka next kita exec dhclient pakai sudo ip netns exec namespace_satu dhclient veth0-peer
• test ping keluar

Setup shadowsocks proxy server

shadowsocks-libev is available in the official repository for Debian 9("Stretch"), unstable, Ubuntu 16.10 and later derivatives:

sudo apt update
sudo apt install shadowsocks-libev

then, create a file config on /etc/shadowsocks-libev/config.json which has contents something like this

{
    "server":["::1", "10.1.1.4"],
    "mode":"tcp_and_udp",
    "server_port":8388,
    "local_port":1080,
    "password":"1u21wW3E0bwu",
    "timeout":86400,
    "method":"chacha20-ietf-poly1305"
}

in this case, I use 10.1.1.4 as local ip, then forwarded through firewall to the public addr.

then, sudo systemctl start shadowsocks-libev

ref: https://shadowsocks.org/doc/configs.html

connect

Alpine htop install on live QEMU

create disk first

qemu-img create -f qcow2 alpine-1.qcow2 1G

start qemu

download iso here https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/x86_64/alpine-virt-3.21.3-x86_64.iso

qemu-system-x86_64 \
      -enable-kvm \
      -m 256 \
      -smp 4 \
      -cpu host \
      -drive file=pc-1.qcow2,format=qcow2 \
      -cdrom alpine-virt-3.21.3-x86_64.iso \
      -boot d \
      -net nic \
      -net user -nographic

default username is root, password left blank, just enter

set eth0 up & DHCP

• ip link set dev eth0 up
• udhcpc -i eth0
• ping 1.1.1.1 (testing)

for /etc/resolv.conf, change to nameserver 1.1.1.1

patch apk configuration

cp /etc/apk/repositories /etc/apk/repositories-old

--- repositories-old
+++ repositories
@@ -1 +1,2 @@
-/media/cdrom/apks
+http://dl-cdn.alpinelinux.org/alpine/v3.21/main
+http://dl-cdn.alpinelinux.org/alpine/v3.21/community

update

• apk update
• apk cache download
• apk cache -v sync

setup partition

make sure first your ext4 kernel module is installed & loaded by typing

• lsmod | grep ext4: check
• modprobe ext4
• lsmod | grep ext4: recheck again

the output is should be something like this

then we can move forward to ext4 partition generation

partitioning

install parted first using apk add parted, check your disk first using fdisk -l, for example there is /dev/sda

parted section

run parted /dev/sda, inside of parted is

• unit s
• mklabel [gpt/msdos]: I use msdos anyway
• print: identify disk size by see on section something like Disk /dev/sda: 2097152s, there has 2097152s sector, so I use 2097152 - (2948 * 2), which is 2093056s
• mkpart
  • primary
  • ext4
  • 2048s
  • 2093056s
    • q

resulted partition:

setup diskless install

warn: you need to enable ext4 kernel module or you'll get mount: mounting /dev/sda1 on /media/sda1 failed: Invalid argument when mounting

• modprobe ext4
• apk add e2fsprogs for mkfs.ext4 command
• mkfs.ext4 /dev/sda1
• mount /dev/sda1 /media/sda1: mount location should be located on /media, not /mnt

then, you can do normal setup-alpine, except for this thing

• Which disk(s) would you like to use? (or '?' for help or 'none') [none] none
• Enter where to store configs ('floppy', 'sda1', 'usb' or 'none') [sda1] sda1
• Enter apk cache directory (or '?' or 'none') [/media/sda1/cache] /media/sda1/cache

LBU

run lbu commit

Run router-OS on qemu

on normal interface

qemu-system-x86_64 \
          -enable-kvm \
          -boot order=d \
          -cdrom chr-6.49.18.img \
          -drive file=chr.qcow2,format=qcow2 \
          -m 4G \
          -smp 4 \
          -net user,hostfwd=tcp::10022-:22,hostfwd=tcp::8291-:8291 \
          -net nic

setup linux bridge between RouterOS & Alpine linux hosts

this docs is second part of this

first thing that we need to do is create a copy of qcow2 alpine, so we do not need to re-install again. do not forget to change the hostname & password for easy identification

setup bridge

• sudo ip link add br0-lan type bridge
• sudo ip link add br1-lan type bridge

setup TAP

tap0 & tap1 connected to br0-lan tap2 & tap3 connected to br1-lan

• sudo ip tuntap add dev tap0 mode tap
• sudo ip tuntap add dev tap1 mode tap
• sudo ip tuntap add dev tap2 mode tap
• sudo ip tuntap add dev tap3 mode tap
• sudo ip link set tap0 master br0-lan
• sudo ip link set tap1 master br0-lan
• sudo ip link set tap2 master br1-lan
• sudo ip link set tap3 master br1-lan
• sudo ip link set tap0 up
• sudo ip link set tap1 up
• sudo ip link set tap2 up
• sudo ip link set tap3 up

launch mikrotik ISO

qemu-system-x86_64 \
      -enable-kvm \
      -boot order=d \
      -drive file=chr.qcow2,format=qcow2 \
      -m 256M \
      -smp 4 \
      -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
      -device virtio-net-pci,netdev=net0,mac=02:11:2a:3b:ff:c3  \
      -netdev tap,id=net1,ifname=tap2,script=no,downscript=no \
      -device virtio-net-pci,netdev=net1,mac=02:1a:f2:f1:21:b5  \
      -nographic \
      -net user,hostfwd=tcp::10022-:22,hostfwd=tcp::8291-:8291 \
      -net nic

based on topology, we know that

• 02:11:2a:3b:ff:c3 used by tap0 (in routerOS, bridge 0)
• 02:1a:f2:f1:21:b5 used by tap2 (in routerOS, bridge 1)
• tap1 & tap3 used by alpine virtual machine

now run alpine iso

alpine (as pc-1)

qemu-system-x86_64 \
            -enable-kvm \
            -m 256 \
            -smp 4 \
            -cpu host \
            -drive file=pc-1.qcow2,format=qcow2 \
            -cdrom alpine-virt-3.21.3-x86_64.iso \
            -boot d \
            -nographic \
            -netdev tap,id=net0,ifname=tap1,script=no,downscript=no \
            -device virtio-net-pci,netdev=net0

alpine (as pc-3)

qemu-system-x86_64 \
            -enable-kvm \
            -m 256 \
            -smp 4 \
            -cpu host \
            -drive file=pc-2.qcow2,format=qcow2 \
            -cdrom alpine-virt-3.21.3-x86_64.iso \
            -boot d \
            -nographic \
            -netdev tap,id=net0,ifname=tap3,script=no,downscript=no \
            -device virtio-net-pci,netdev=net0

WinBox config

first, we need to see the configuration & which etherX by looking to spesific mac-addr, we find out that

[admin@MikroTik] > /interface print
Flags: D - dynamic, X - disabled, R - running, S - slave 
 #     NAME                                TYPE       ACTUAL-MTU L2MTU  MAX-L2MTU MAC-ADDRESS      
 0  R  ;;; wan
       ether1                              ether            1500                  52:54:00:12:34:56
 1  R  ether4                              ether            1500                  02:11:2A:3B:FF:C3
 2  R  ether5                              ether            1500                  02:1A:F2:F1:21:B5
[admin@MikroTik] > 

next thing, lets configure our ip using winbox, this is ip configuration

DHCP server section

now, try run udhcpc -i eth0 on each vm

result on pc-1

result on pc-2

Linux iptables preview (try making a nat)

for example, iptables -t nat the t option from manpage is

          filter:
              This  is  the  default  table  (if  no -t option is passed). It contains the built-in
              chains INPUT (for packets destined to local  sockets),  FORWARD  (for  packets  being
              routed through the box), and OUTPUT (for locally-generated packets).

          nat:
              This  table  is consulted when a packet that creates a new connection is encountered.
              It consists of four built-ins: PREROUTING (for altering packets as soon as they  come
              in),  INPUT  (for  altering packets destined for local sockets), OUTPUT (for altering
              locally-generated packets before routing), and POSTROUTING (for altering  packets  as
              they are about to go out).  IPv6 NAT support is available since kernel 3.7.

          mangle:
              This table is used for specialized packet alteration.  Until kernel 2.4.17 it had two
              built-in chains: PREROUTING (for altering incoming packets before routing) and OUTPUT
              (for  altering locally-generated packets before routing).  Since kernel 2.4.18, three
              other built-in chains are also supported: INPUT (for packets coming into the box  it‐
              self),  FORWARD  (for altering packets being routed through the box), and POSTROUTING
              (for altering packets as they are about to go out).

          raw:
              This table is used mainly for configuring exemptions from connection tracking in com‐
              bination with the NOTRACK target.  It registers at the netfilter  hooks  with  higher
              priority and is thus called before ip_conntrack, or any other IP tables.  It provides
              the  following  built-in chains: PREROUTING (for packets arriving via any network in‐
              terface) and OUTPUT (for packets generated by local processes).

          security:
              This table is used for Mandatory Access Control (MAC) networking rules, such as those
              enabled by the SECMARK and CONNSECMARK targets.  Mandatory Access Control  is  imple‐
              mented by Linux Security Modules such as SELinux.  The security table is called after
              the filter table, allowing any Discretionary Access Control (DAC) rules in the filter
              table  to  take  effect before MAC rules.  This table provides the following built-in
              chains: INPUT (for packets coming into the box itself), OUTPUT (for altering locally-
              generated packets before routing), and FORWARD (for  altering  packets  being  routed
              through the box).

What kind of tables in iptables

according from archlinux wiki

iptables contains five tables:

• raw is used only for configuring packets so that they are exempt from connection tracking.
• filter is the default table, and is where all the actions typically associated with a firewall take place.
• nat is used for network address translation (e.g. port forwarding).
• mangle is used for specialized packet alterations.
• security is used for Mandatory Access Control networking rules (e.g. SELinux -- see this article for more details).

In most common use cases, you will only use two of these: filter and nat. The other tables are aimed at complex configurations involving multiple routers and routing decisions and are in any case beyond the scope of these introductory remarks.

Tables & chain details

all iptables chain information can be gathered by this command

sudo iptables -t filter -L
sudo iptables -t nat -L
sudo iptables -t mangle -L
sudo iptables -t raw -L
sudo iptables -t security -L

this is list of all chain by corresponding table

table raw

• PREROUTING
• OUTPUT

table nat

• PREROUTING
• INPUT
• OUTPUT
• POSTROUTING

table mangle

• PREROUTING
• INPUT
• FORWARD
• OUTPUT
• POSTROUTING

table filter

• INPUT
• FORWARD
• OUTPUT

table security

• INPUT
• FORWARD
• OUTPUT

we will focus on NAT section.

machine session

In iptables, packet often categorized as 4 different state, such

• NEW
• ESTABLISHED
• RELATED
• INVALID

this connection tracking is done by a special framework within the kernel called conntrack

command lists

this is some special iptables command collection

• iptables --list-rules: show all ip rules
• iptables --table nat --list --line-numbers: will useful if you want to delete spesific rule, i.e, duplicated rule
• sudo iptables --table nat -D POSTROUTING 2 example: delete rule 2

create your own chain

• iptables -N chain_name: eq: --new-chain
• iptables -A chain_name -p icmp -j accept (example)

make changes permanent

sudo iptables-save > /etc/iptables/rules.v4

tables explanation

> sudo iptables --table nat --list
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  anywhere             anywhere            
MASQUERADE  all  --  anywhere             anywhere            
MASQUERADE  all  --  anywhere             anywhere            
MASQUERADE  all  --  anywhere             anywhere            

target: what to do with packet. prot: abbr of protocol opt: abbr of options (AI says rarely used) source: its anywhere or explicit ip such 192.168.1.0/24 destination: same as source, can be network

show rules from X table

for example, there has

• nat table, which contains PREROUTING, INPUT, OUTPUT, POSTROUTING
• filter table contains main chain that used by iptables for packet filtering, which contains INPUT, FORWARD, OUTPUT

MORE deeper on X table

Linux QEMU network address translation

first all, set setup our interface

• sudo ip tuntap add tap0 mode tap
• sudo ip tuntap add tap1 mode tap (optional, if there has any more vm)
• sudo ip link add br0-lan type bridge (our switch)
• sudo ip addr add 192.168.12.1/24 dev br0-lan
• sudo ip link set br0-lan up
• sudo ip link set tap0 up, also with tap1 if needed

Run alpine guest hosts

qemu-system-x86_64 \
            -enable-kvm \
            -m 256 \
            -smp 4 \
            -cpu host \
            -drive file=pc-2.qcow2,format=qcow2 \
            -cdrom alpine-virt-3.21.3-x86_64.iso \
            -boot d \
            -nographic \
            -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
            -device virtio-net-pci,netdev=net0

Inside of guest VM

Your ip addr output might something like this

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe12:3456/64 scope link 
       valid_lft forever preferred_lft forever

now set eth0 up by typing

• ip link set eth0 up
• ip addr add 192.168.12.2/24 dev eth0 (I manually setting up IP)

check ip route

pc-2:~# ip r
192.168.12.0/24 dev eth0 scope link  src 192.168.12.2 

there is no route, run ip route add default via 192.168.12.1, check again with ip route

pc-2:~# ip r
default via 192.168.12.1 dev eth0 
192.168.12.0/24 dev eth0 scope link  src 192.168.12.2 

setting up firewall

we want everyting from br0-lan is forwarded into wlan0, vice versa. in order to do that, we need NAT (network address translation)

here

check your nat table first

sudo iptables --table nat --list -v, make sure there is no

 2151  499K MASQUERADE  all  --  any    wlan0   anywhere             anywhere            

now run

• sudo iptables --table nat --append POSTROUTING --out-interface wlan0 -j MASQUERADE
• sudo iptables -t filter -A FORWARD -i wlan0 -o br0-lan -m state --state RELATED,ESTABLISHED -j ACCEPT
• sudo iptables -t filter -A FORWARD -i br0-lan -o wlan0 -j ACCEPT

QEMU block 8.8.8.8 from host

after this

I want this rule

• allow other ICMP (i.e 1.1.1.1)
• deny ICMP 8.8.8.8

iptables rule

suppose, vm ip is 192.168.12.2, so the iptables command is

sudo iptables -t filter -A FORWARD -s 192.168.12.2 -d 8.8.8.8 -p icmp -j DROP

note:

• t: explicit table
• a: table FORWARD
• s: source from 192.168.12.2 (OUR VM)
• d: destination: 8.8.8.8
• p: protocol (which is ICMP)
• j: jump

the problem

you may see ping 8.8.8.8 still success, let check iptables rule

sudo iptables -t filter -L FORWARD -n --line-numbers -v

the output

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
2    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
3    DROP       icmp --  192.168.12.2         8.8.8.8             

see? when ping to 8.8.8.8, its stop at rule 2, which abort 3st rule, in order to fix it, we need re-order this

remote all rule

sudo iptables -t filter -F FORWARD

re-add

sudo iptables -t filter -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -t filter -A FORWARD -s 192.168.12.2 -d 8.8.8.8 -p icmp -j DROP
sudo iptables -t filter -A FORWARD -s 192.168.12.2/24 -j ACCEPT

Try re-run sudo iptables -t filter -L FORWARD -n --line-numbers

the order should be fixed

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         
1               all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
2    DROP       icmp --  192.168.12.2         8.8.8.8             
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           

logic netfilter linux kernel

Logic NAT

misalkan kita ada virtual interface br0, dengan anggotanya tap0, tap1 lalu ingin forward semua traffic ke wlan0 (karna memang wlan0 tidak bisa di jadikan master dari x y z)

maka solusinya iptables

sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE

logic nya

• fill dulu tabel nat
• append ke postrouting, jadi setelah packet di routing oleh linux kernel, ada 2 field, source addr (katakanlah ip tap0) lalu destination addr (misal 8.8.8.8)
• nah, semisal tidak di apa apakan, packet sent ke 8.8.8.8, tapi 8.8.8.8 tidak bisa mengirim back packetnya karna source addr nya invalid (di stage ini, source ip nya adalah IP LAN)
• maka, setelah linux kernel meng-routingkan packet nya, seketika setelah itu (postrouting), kita ganti source addrnya dengan source ip PUBLIK milik firewall, jadi 8.8.8.8 bisa tahu kemana packet harus dikirim kembali
• didalam firewall, packet yang tadi diterima, direverse balik oleh nat, dan dikirim balik ke perangkat aslinya.
• pakai MASQUERADE

logic forward data dari br0 ke wlan

sudo iptables -A FORWARD -i br0 -o wlan0 -j ACCEPT

important notes:

• chain FORWARD: This chain is only used for packets that are not destined for the local machine, but are routed through it (i.e., from one interface to another).
• In this case: a device connected to br0 (e.g., a VM or container) wants to access the internet via wlan0.

analoginya

[Device 192.168.100.2] ──> [br0 (Linux bridge)] ──> Linux router ──> [wlan0] ──> Internet

bisa juga -j diganti drop, maka aliran data akan terputus

logic forward dari wlan0 ke br0 (sebaliknya)

sudo iptables -A FORWARD -i wlan0 -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT

logic:

• misal vm tap0 buat koneksi keluar, misal ke 8.8.8.8
• dari tap0 -> br0 -> wlan0 di allow oleh rule sudo iptables -A FORWARD -i br0 -o wlan0 -j ACCEPT
• ketika server mengirim respon balik, maka alurnya dari wlan0 ke br0, nah ini fungsi dari sudo iptables -A FORWARD -i wlan0 -o br0 -j ACCEPT

KENAPA harus pakai --state RELATED,ESTABLISHED karna kita butuh packet yg sudah estab duluan yg boleh lewat. ini akan menghindari jaringan luar (wlan0) mengakses br0 -> tap0 & tap1

resep nginx oleh pak hanif

cuplikan config nginx di server, cc @hansputera

server {
        listen 80; # tanpa server_default
        listen [::]:80;

        root /var/www/html/blablabla.fadev.org;
        
        index index.php index.html index.htm index.nginx-debian.html;

        server_name blablabla.fadev.org;

        location ~ \.php$ {
                include snippets/fastcgi-php.conf;

                fastcgi_pass unix:/run/php/php-fpm.sock;
        }
}

ip command

ip addr

ip addr is used to display and manage IP addresses assigned to network interfaces.

example:

• ip addr
• ip addr show
• sudo ip addr add 192.168.1.100/24 dev eth0
• sudo ip addr del 192.168.1.100/24 dev eth0

anything about ip assigment

ip link

ip link is used to display and manage network interfaces (links), focusing on layer 2 (Ethernet, loopback, etc.), not IP addresses.

example:

• ip link
• sudo ip link set dev eth0 up
• sudo ip link set dev eth0 down
• sudo ip link set dev eth0 mtu 1400

anyting about interface

ip tuntap

ip tuntap manages TUN/TAP virtual network interfaces, which provide a way for user-space programs to interact with network packets as if they were network devices.

• TUN: Simulates a point-to-point device (layer 3), used for routing IP packets (e.g., VPN tunnels).
• TAP: Simulates an Ethernet device (layer 2), used for bridging Ethernet frames (e.g., virtual switches, VMs).

example:

• sudo ip tuntap add mode tap dev tap0
• sudo ip tuntap add mode tun dev tun0
• sudo ip tuntap del mode tap dev tap0

ip route

IP routing is the process by which data packets are directed from their source to their destination across interconnected networks using Internet Protocol (IP). This process is fundamental to the operation of the Internet and all IP-based networks.

• https://www.link11.com/en/glossar/ip-routing/
• https://www.cloudflare.com/learning/network-layer/what-is-routing/
• https://info.support.huawei.com/info-finder/encyclopedia/en/IP+routing.html

View routing table:

• ip route

add route

• ip route add <destination> via <gateway> dev <interface>

example:

pc-1:~# ip route add default via 192.168.15.1 dev eth0
pc-1:~# ip route
default via 192.168.15.1 dev eth0 
192.168.15.0/24 dev eth0 scope link  src 192.168.15.2 

delete route

• sudo ip route del <destination> [via <gateway>] [dev <interface>]

example:

pc-1:~# ip route delete default via 192.168.15.1 dev eth0
pc-1:~# ip r
192.168.15.0/24 dev eth0 scope link  src 192.168.15.2 
pc-1:~# 

qemu default network

The default gateway in QEMU when using the default user-mode networking (SLIRP) is typically 10.0.2.2 on the network 10.0.2.0/24. QEMU creates a virtual NAT network where the guest is assigned an IP like 10.0.2.15 by DHCP, and the gateway for the guest to reach outside networks is 10.0.2.2. This gateway is implemented inside QEMU and handles routing and NAT for the guest's traffic.

• Default network: 10.0.2.0/24
• Default gateway: 10.0.2.2
• Default guest IP (DHCP assigned): 10.0.2.15
• Default DNS server in guest: 10.0.2.3

This setup requires no special host configuration or root privileges and is the default networking mode for QEMU user mode networking.

additional

• https://docs.zephyrproject.org/latest/connectivity/networking/qemu_user_setup.html
• https://darnassus.sceen.net/~hurd-web/hurd/running/qemu/networking/
• https://wiki.qemu.org/Documentation/Networking

mikrotik firewall

each firewall module (called as table in iptables) has its own predefined chain

• raw: dipakai sebelum data dilihat oleh conntrack, NAT, masuk routing, dll. intinya di sinilah proses early sebelum benar2 di proses. karna belum sampai ke conntrack, maka akan sangat fast, dipakai untuk mitigasi ddos

  karna statenya ada sebelum conntrack, maka tidak ada status kayak estab, dll, tapi sangat fast + low cpu, only raw matching

  Karna raw table melihat packet sebelum NAT, maka

  • Destination IPs are still the original IPs from the client
  • Source IPs haven't been changed by masquerade or src-nat yet

  chain chain nya

  • prerouting: packet yang datang dari luar just sebelum masuk conntrack
  • output: packet yang digenerate oleh aplikasi, juga statenya sebelum di track oleh conntrack

  contoh

  • /ip firewall raw add chain=prerouting src-address=8.8.8.8 action=drop

  tambahan:

  letak raw di packet flow

  IN → RAW → MANGLE (pre) → CONNECTION TRACKING → NAT (dstnat) → FILTER (input/forward) → MANGLE (post) → NAT (srcnat) → OUT
  

• filter

  • input
  • forward
  • output

• mangle

  • prerouting
  • input
  • forward
  • output
  • postrouting

• nat

  • srcnat
  • dstnat

chains

RouterOS consist of a few default chains. These chains allow you to filter packets at various points:

• The PREROUTING chain: Rules in this chain apply to packets as they just arrive on the network interface. This chain is present in the nat, mangle and raw tables.
• The INPUT chain: Rules in this chain apply to packets just before they’re given to a local process. This chain is present in the mangle and filter tables.
• The OUTPUT chain: The rules here apply to packets just after they’ve been produced by a process. This chain is present in the raw, mangle, nat, and filter tables.
• The FORWARD chain: The rules here apply to any packets that are routed through the current host. This chain is only present in the mangle and filter tables.
• The POSTROUTING chain: The rules in this chain apply to packets as they just leave the network interface. This chain is present in the nat and mangle tables.

bring wlan0 connectivity to the network namespace linux & test routerOS

setup network namespace

• sudo ip netns add firefoxns
• sudo ip netns exec firefoxns ip link add br0-lan type bridge
• sudo ip netns exec firefoxns ip tuntap add tap0 mode tap
• sudo ip netns exec firefoxns ip tuntap add tap1 mode tap
• sudo ip netns exec firefoxns ip tuntap add tap2 mode tap
• sudo ip netns exec firefoxns ip link set dev tap0 master br0-lan
• sudo ip netns exec firefoxns ip link set dev tap1 master br0-lan
• sudo ip netns exec firefoxns ip link set dev tap2 master br0-lan

enable the network interface (inside netns)

• sudo ip netns exec firefoxns ip link set br0-lan up
• sudo ip netns exec firefoxns ip link set tap0 up
• sudo ip netns exec firefoxns ip link set tap1 up
• sudo ip netns exec firefoxns ip link set tap2 up

create virtual eth pair between host & netns

• sudo ip link add veth0 type veth peer name veth0-peer
• sudo ip link set veth0-peer netns firefoxns

assign ip for virtual eth & virtual eth peer

• sudo ip addr add 10.200.1.1/24 dev veth0
• sudo ip netns exec firefoxns ip addr add 10.200.1.2/24 dev veth0-peer
• sudo ip link set veth0 up
• sudo ip netns exec firefoxns ip link set veth0 up

setup routing table

• sudo ip netns exec firefoxns ip route add default via 10.200.1.1

setup ip forwarding & network address translation

• sudo sysctl -w net.ipv4.ip_forward=1
• sudo iptables -t nat -A POSTROUTING -s 10.200.1.0/24 -o wlan0 -j MASQUERADE

test

• sudo ip netns exec ping 1.1.1.1

now, run routeros & alpine inside of netns

routeros

sudo ip netns exec firefoxns qemu-system-x86_64 \
            -enable-kvm \
            -m 256 \
            -smp 4 \
            -cpu host \
            -drive file=chr.qcow2,format=qcow2 \
            -boot d \
            -nographic \
            -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
            -device virtio-net-pci,netdev=net0,mac=02:aa:bb:cc:dd:ee \
            -net user,hostfwd=tcp::8291-:8291 \
            -net nic

alpine

sudo ip netns exec firefoxns qemu-system-x86_64 \
            -enable-kvm \
            -m 256 \
            -smp 4 \
            -cpu host \
            -drive file=pc-2.qcow2,format=qcow2 \
            -cdrom alpine-virt-3.21.3-x86_64.iso \
            -boot d \
            -nographic \
            -netdev tap,id=net0,ifname=tap2,script=no,downscript=no \
            -device virtio-net-pci,netdev=net0,mac=$(randommac)

network namespace + NAT notes

• ip a
• sudo ip link add veth0 type veth peer name veth0-peer
• sudo ip addr add 10.200.1.1/24 dev veth0
• sudo ip netns add firefoxns
• sudo ip link set veth0-peer netns firefoxns
• sudo sysctl -w net.ipv4.ip_forward=1
• sudo iptables -t nat -A POSTROUTING -s 10.200.1.0/24 -o wlan0 -j MASQUERADE
• sudo ip netns exec firefoxns ip addr add 10.200.1.2/24 dev veth0-peer
• sudo ip netns exec firefoxns ip addr add 127.0.0.1/8 dev lo
• sudo ip netns exec firefoxns ip link set lo up
• sudo ip netns exec firefoxns ip link set veth0-peer up
• sudo ip netns exec firefoxns ip route add default via 10.200.1.1
• sudo ip netns exec firefoxns ip link add br0-lan type bridge
• sudo ip netns exec firefoxns ip tuntap add tap0 mode tap
• sudo ip netns exec firefoxns ip tuntap add tap1 mode tap
• sudo ip netns exec firefoxns ip tuntap add tap2 mode tap
• sudo ip netns exec firefoxns ip link set dev tap0 master br0-lan
• sudo ip netns exec firefoxns ip link set dev tap1 master br0-lan
• sudo ip netns exec firefoxns ip link set dev tap2 master br0-lan
• sudo ip netns exec firefoxns ip link set dev tap0 up
• sudo ip netns exec firefoxns ip link set dev tap1 up
• sudo ip netns exec firefoxns ip link set dev tap2 up
• sudo ip netns exec firefoxns ip link set dev br0-lan up
• sudo ip netns exec firefoxns qemu-system-x86_64 -enable-kvm -m 256 -smp 4 -cpu host -drive file=chr.qcow2,format=qcow2 -boot d -nographic -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -device virtio-net-pci,netdev=net0,mac=02:aa:bb:cc:dd:ee -net user,hostfwd=tcp::8291-:8291 -net nic
• sudo ip netns exec firefoxns qemu-system-x86_64 -enable-kvm -m 256 -smp 4 -cpu host -drive file=pc-1.qcow2,format=qcow2 -cdrom alpine-virt-3.21.3-x86_64.iso -boot d -nographic -netdev tap,id=net0,ifname=tap1,script=no,downscript=no -device virtio-net-pci,netdev=net0,mac=$(randommac)

randommac source code

use rand::Rng;

fn main() {
    let mut rng = rand::thread_rng();
    let mac = format!(
        "52:54:00:{:02x}:{:02x}:{:02x}",
        rng.gen::<u8>(),
        rng.gen::<u8>(),
        rng.gen::<u8>()
    );
    println!("{}", mac);
}

deploy Docmost on server

please read this doc first, https://docmost.com/docs/installation

docker compose config

version: "3"

services:
  docmost:
    image: docmost/docmost:latest
    depends_on:
      - db
      - redis
    environment:
      APP_URL: "http://localhost:3000"
      APP_SECRET: "REPLACE_WITH_LONG_SECRET"
      DATABASE_URL: "postgresql://docmost:STRONG_DB_PASSWORD@db:5432/docmost?schema=public"
      REDIS_URL: "redis://redis:6379"
    ports:
      - "3000:3000"
    restart: unless-stopped
    volumes:
      - docmost:/app/data/storage

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: docmost
      POSTGRES_USER: docmost
      POSTGRES_PASSWORD: STRONG_DB_PASSWORD
    restart: unless-stopped
    volumes:
      - db_data:/var/lib/postgresql/data

  redis:
    image: redis:7.2-alpine
    restart: unless-stopped
    volumes:
      - redis_data:/data

volumes:
  docmost:
  db_data:
  redis_data:

replace your STRONG_DB_PASSWORD with openssl rand -hex 10, and REPLACE_WITH_LONG_SECRET with openssl rand -hex 32, and APP_URL, for example, in my case is https://docmost.fadev.org

run docker compose run

setup reverse proxy (nginx)

go to /etc/nginx/sites-enabled, and create a clone with cp default docmost.fadev.org

fill it like this

server_name docmost.fadev.org;

location / {
	# First attempt to serve request as file, then
	# as directory, then fall back to displaying a 404.
	#try_files $uri $uri/ =404;

	proxy_pass http://127.0.0.1:3000;
}

location /socket.io/ {
	proxy_pass http://127.0.0.1:3000;
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "upgrade";
	proxy_set_header Host $host;
}

location /collab {
	proxy_pass http://127.0.0.1:3000;
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "upgrade";
	proxy_set_header Host $host;
}

HTTPS

run

• sudo apt install certbot python3-certbot-nginx
• sudo certbot --nginx -d docmost.fadev.org

Firewall

allow port 443

fix vthxxxxxxxx is not connected to docker0

consider

> ip  a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 2c:4d:54:c6:f2:0c brd ff:ff:ff:ff:ff:ff
    altname enx2c4d54c6f20c
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f0:03:8c:66:9c:21 brd ff:ff:ff:ff:ff:ff
    inet 192.168.105.162/24 metric 1024 brd 192.168.105.255 scope global dynamic wlan0
       valid_lft 2920sec preferred_lft 2920sec
    inet 192.168.105.163/24 brd 192.168.105.255 scope global secondary dynamic noprefixroute wlan0
       valid_lft 2921sec preferred_lft 2921sec
    inet6 fe80::ee67:75f9:fa9c:2561/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 86:1e:e0:72:f4:8d brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::841e:e0ff:fe72:f48d/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
17: veth914db4f@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether d2:02:61:f4:12:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::d002:61ff:fef4:12d0/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

you can see, veth914db4f is not connected to br0. let's try hard

• run image (explicitly) as bridge network: docker run --network bridge -it --rm -p 127.0.0.1:8000:4000 php-test-server (this is nothing happen)
• force connect: docker network connect bridge 2cf520235e6a (this is show Error response from daemon: endpoint with name mystifying_yonath already exists in network bridge, but ip a says vethxxxx is not master to anyone). you can try disconnect & connect again

also

docker inspect -f '{{json .NetworkSettings.Networks}}' 2cf520235e6a | jq
{
  "bridge": {
    "IPAMConfig": null,
    "Links": null,
    "Aliases": null,
    "MacAddress": "5a:2d:5d:7c:c1:86",
    "DriverOpts": null,
    "GwPriority": 0,
    "NetworkID": "08f151565c95ba052f682c7560e55199e2d75f3d2348af8f98a9711e9294b3fd",
    "EndpointID": "0522c15ad18fbae05071a7d5e09944b7777deaafd3490082cac237fdfcd14f0c",
    "Gateway": "172.17.0.1",
    "IPAddress": "172.17.0.2",
    "IPPrefixLen": 16,
    "IPv6Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "DNSNames": null
  }
}

the medicine: https://forums.docker.com/t/no-connection-to-the-bridge-by-default/134619/8

sudo systemctl stop systemd-networkd.service                                                    
sudo systemctl disable systemd-networkd.service                                                        
sudo systemctl stop systemd-networkd.socket                                                            
sudo systemctl disable systemd-networkd.socket
sudo systemctl start NetworkManager
sudo systemctl enable NetworkManager

php-fpm config

file ini biasanya ada di

• /usr/local/etc/php-fpm.d/www.conf (di docker php-fpm)
• /etc/php/8.3/fpm/pool.d/www.conf (di debian bullseye)

intinya, tidak pasti haha.

config structure

general structure nya seperti ini kalau ';' dihilangkan

[www]
user = www-data
group = www-data
listen = /run/php/php8.3-fpm.sock
listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

config global (WIP)

lokasi config ini ada di

/etc/php/php-fpm.conf

config untuk pool (each)

best read https://www.php.net/manual/en/install.fpm.configuration.php

• [pool_name] intinya ini unique, tidak ada yang sama
• user / group
  • user: dimana child process akan di run (as user?)
  • group: liat /etc/passwd
• listen

  • listen: dimana daemon akan listen yg nanti akan di reverse proxy oleh nginx, nilai defaultnya biasanya /run/php/php8.3-fpm.sock atau 127.0.0.1:9000, contoh:

    • ip.add.re.ss:port
    • [ip:6:addr:ess]:port
    • port
    • /path/to/unix/socket

  • listen.backlog: jumlah queue pending connection yang bisa di hold

  • listen.owner: configure ke mana kah /run/php/php8.3-fpm.sock itu ownernya

  • listen.group: same as listen.owner

  • listen.mode: nomor permission si /run/php/php8.3-fpm.sock listen

  • listen.acl_users: WIP

  • listen.acl_groups: WIP

  • listen.allowed_clients: set dari mana FCGI boleh diakses, di kasus nginx, mostly 127.0.0.1, except docker, dia pakai br-xxxxxx yang ip nya pasti bukan 127.0.0.1

  • listen.setfib: WIP

systemd config template

root location: /etc/systemd/system/<YOUR_SERVICE>.service

template

[Unit]
Description=Random service
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=root
ExecStart=/usr/bin/randomprogram -c /etc/randomprogram/other.toml

[Install]
WantedBy=multi-user.target

setting up ubuntu server with qemu + custom bridge

qemu section

# create image first
qemu-img create -f qcow2 ubuntu-server.img 10G

# run the iso
qemu-system-x86_64 \
      -enable-kvm \
      -boot order=d \
      -cdrom ubuntu-24.04.2-live-server-amd64.iso \
      -drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd \
      -drive if=pflash,format=raw,file=OVMF_VARS_ubuntu_server_gpt.4m.fd \
      -drive file=ubuntu-server.img,format=qcow2 \
      -m 4G \
      -smp 4 \
      -net user,hostfwd=tcp::20022-:22 \
      -net nic

# run iso
qemu-system-x86_64 \
	-enable-kvm \
	-boot order=d \
	-drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd \
	-drive if=pflash,format=raw,file=OVMF_VARS_ubuntu_server_gpt.4m.fd \
	-drive file=ubuntu-server.img,format=qcow2 \
	-m 4G \
	-smp 4 \
	-net user,hostfwd=tcp::20022-:22 \
	-net nic \
	-vga virtio

setting up routeros + ubuntu server (freeradius) + and archlinux client

first, setup virtual lan

setup configurasi ip hingga seperti ini

51: br0-lan: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 8a:b9:27:b3:c7:a5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::88b9:27ff:feb3:c7a5/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
53: tap0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br0-lan state DOWN group default qlen 1000
    link/ether c6:33:f3:fe:d3:ba brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c433:f3ff:fefe:d3ba/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
54: tap1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br0-lan state DOWN group default qlen 1000
    link/ether a2:eb:ee:56:8a:90 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0eb:eeff:fe56:8a90/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
55: tap2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master br0-lan state DOWN group default qlen 1000
    link/ether c6:c7:1c:7c:67:8a brd ff:ff:ff:ff:ff:ff

untuk membuatnya, jalankan skrip command command ini:

• sudo ip link add br0-lan type bridge <- buat switch virtual
• sudo ip tuntap add tap0 mode tap <- buat colokan virtual
• sudo ip tuntap add tap1 mode tap
• sudo ip tuntap add tap2 mode tap
• sudo ip link set tap0 master br0-lan <- set semua colokan punya induk ke switch br0-lan, intinya harus satu switch
• sudo ip link set tap1 master br0-lan
• sudo ip link set tap2 master br0-lan
• sudo ip link set br0-lan up <- nyalakan semua interface nya
• sudo ip link set tap0 up
• sudo ip link set tap1 up
• sudo ip link set tap2 up

alokasi interface

• br0-lan --> switch utama
• tap0 --> interface untuk routerOS
• tap1 --> interface untuk freeradius (running di ubuntu server)
• tap2 --> interface test client (bisa alpineLinux, archlinux, bebas)

run routeros

download links: https://download.mikrotik.com/routeros/6.49.18/chr-6.49.18.img.zip

# convert first
qemu-img convert -f raw -O qcow2 chr-6.49.18.img chr.qcow2

# then run
qemu-system-x86_64 \
      -enable-kvm \
      -smp 4 \
      -m 256M \
      -drive file=chr.qcow2,format=qcow2 \
      -boot order=d \
      -net user,hostfwd=tcp::8291-:8291 \
      -net nic \
      -nographic \
      -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
      -device virtio-net-pci,netdev=net0,mac=02:11:2a:3b:ff:c3

setidaknya, mac-addr nya sama. yaitu ether3

run ubuntu server

command ini hanya dirun run sekali (ketika instalasi)

# copy firmware uefi vars
cp /usr/share/edk2/x64/OVMF_VARS.4m.fd .
mv OVMF_VARS.4m.fd OVMF_VARS_ubuntu_server_gpt.4m.fd

# run qemu
qemu-system-x86_64 \
    -enable-kvm \
    -boot order=d \
    -cdrom ubuntu-24.04.2-live-server-amd64.iso \
    -drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd \
    -drive if=pflash,format=raw,file=OVMF_VARS_ubuntu_server_gpt.4m.fd \
    -drive file=ubuntu-server.img,format=qcow2 \
    -m 4G \
    -smp 4 \
    -net user,hostfwd=tcp::20022-:22 \
    -net nic

proses instalasi ubuntu server cari di internet xixi.

setelah installed, kita install beberapa base package dahulu. base package yg akan diinstall adalah

sudo apt-get update && \
sudo apt-get upgrade && \
sudo apt-get install \
    openssh-server freeradius isc-dhcp-client

lalu setelah installed, kita tidak butuh iso lagi, kita bisa buang saja parameter -cdrom ubuntu-24.04.2-live-server-amd64.iso karna unused. dan untuk konfigurasi tambahan, saya buat pakai vga virtio (agar text nya smooth), dan juga netdev untuk colokan tap1 yang nyambung ke bridge

qemu-system-x86_64 \
    -enable-kvm \
    -boot order=d \
    -drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd \
    -drive if=pflash,format=raw,file=OVMF_VARS_ubuntu_server_gpt.4m.fd \
    -drive file=ubuntu-server.img,format=qcow2 \
    -m 4G \
    -smp 4 \
    -netdev user,id=net0,hostfwd=tcp::20022-:22 \
    -device e1000,netdev=net0 \
    -netdev tap,id=net1,ifname=tap1,script=no,downscript=no \
    -device virtio-net-pci,netdev=net1,mac=02:11:2a:3b:aa:c4 \
    -vga virtio

nb: jangan lupa sudo apt-get update && sudo apt-get upgrade && sudo apt-get install openssh-server, karna di vm tidak bisa copy paste (mau dari dalam ataupun keluar), dan juga tidak bisa scroll atas bawah. maka proses me-config nya akan pakai ssh saja

intinya biarkan vm nya jalan begitu saja, bisa di minimize atau dipindah ke tab lain yang tidak menganggu. selanjutnya remote vm nya pakai

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null  fadhil_riyanto@127.0.0.1 -p 20022

nb: ganti fadhil_riyanto dengan username yang sudah diset tadi

setidaknya seperti itu remote-an nya

freeradius

jalankan sudo apt-get install freeradius, lalu test dengan sudo freeradius -X

pastikan ada tulisan ready to process request

winbox

karna winbox pakai port 8291 untuk merepote mikrotik, maka kita konek dengan host 127.0.0.1:8291, password sesuai config awal mikrotik ketika awal tadi

winbox localhost

setup mikrotik dhcp server

ip -> address

ip -> dhcp server

dns server kita biarkan dahulu

pembuktian: jalankan sudo dhclient enp4 (karna interface tun1 dia dibaca sebagai enp4 di ubuntu server)

ping ke router

router ke ubuntu server

setting up Archlinux (GUI)

download dahulu bahan nya di https://sourceforge.net/projects/arch-linux-gui/, pakai xfce (paling ringan)

run archlinux gui sebagai tap2

qemu-system-x86_64 \
    -enable-kvm \
    -boot order=d \
    -cdrom archlinux-gui-xfce-2022.07-x86_64.iso \
    -m 4G \
    -smp 4 \
    -netdev tap,id=net1,ifname=tap2,script=no,downscript=no \
    -device virtio-net-pci,netdev=net1,mac=02:11:5c:1b:aa:c9 \
    -vga virtio

catatan: macnya dikarang lagi, agar tidak sama. jika sama ternyata bakal dapat ip yang sama

nb: archlinux makan 2 ip karna NetworkManager dan systemd-resolved bentrok (bawaannta sepertinya). tapi harusnya no problem at all

setup hotspot winbox

ip -> hotspot

next next saja, kecuali dns nanti diisi 1.1.1.1, dan dns-name diisi nama domain untuk captive portalnya

test

karna 192.168.1.254 ip freeradius

maka kita ignore pakai

ip binding

static ip versions

• sudo ip addr add 192.168.1.2/24 dev ens4
• sudo ip link set dev ens4 up
• sudo ip route append default via 192.168.1.1

freeradius

ubah /etc/freeradius/3.0/clients.conf, tambah

client private-network-1 {
    ipaddr      = 192.192.1.1/24
    secret      = testing123
}

lalu /etc/freeradius/3.0/users

bob Cleartext-Password := "password"

winbox

system -> AAA

SQL authentication freeradius server

first, desclaimer. I use Ubuntu 24.04.2 LTS that running on Archlinux qemu host. so this is not host machine. big thanks for linux netdev that makes local networking possible.

first, lets configure postgresql server

root@integral2:/home/fadhil_riyanto# su postgres
postgres@integral2:/home/fadhil_riyanto$ psql
psql (16.9 (Ubuntu 16.9-0ubuntu0.24.04.1))
Type "help" for help.

postgres=# 

database setup

get it inside. then run

do not forget to allow CRUD option on all tables by running

\c radius
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO radius;

radius server configuration

the sequence of config file

• /etc/freeradius/3.0/clients.conf <-- specify 'who' can connect, we need define it in this file
• /etc/freeradius/3.0/users <-- [manual authentication, using file]
• /etc/freeradius/3.0/radiusd.conf:986 <-- this file enable sql module.
• /etc/freeradius/3.0/mods-available/sql <-- this file is where we configure user/pass for sql module
• /etc/freeradius/3.0/sites-available/default <-- sequence layer of login,
  • first, we filter-out all args
  • then go to preprocess
  • send to log
  • chap & mschap
  • check whatever has digest
  • check whatever user@domain format
  • eap
  • files
  • lookup to sql <-- uncomment this

as root

• /etc/freeradius/3.0/mods-enabled
• ln -s ../mods-available/sql ./sql
• uncomment this line in radiusd.conf
• change this line to postgresql (mods-enabled/sql)

also

• fill the pgsql server credentials (/etc/freeradius/3.0/mods-config/sql/main/postgresql/queries.conf) (aka raddb/sql/database/dialup.conf in rhel)

when connection failed

when connection succeed

populating

best read: https://wiki.freeradius.org/guide/SQL-HOWTO#populating-sql

big thanks for freeradius developer, the sql file is located at /etc/freeradius/3.0/mods-config/sql/main/postgresql/schema.sql

import it with:

psql -U radius -d radius -f /etc/freeradius/3.0/mods-config/sql/main/postgresql/schema.sql

configuration

lets look at sites-available/default

this configuration only allow authorize users that come from NAS-Port-Type == "Wireless-802.11", this is the difference

footage 1: from winbox request

this one come from captive portal

Linux netplan

ref: https://documentation.ubuntu.com/server/explanation/networking/configuring-networks/

example

network:
  version: 2
  ethernets:
    ens3:
      dhcp4: true
    ens4:
      addresses:
        - 192.168.1.2/24
      routes:
        - to: default
          via: 192.168.1.1

location: /etc/netplan note: execution is ordered by file name

then run sudo netplan generate

sshfs

the setup is very easy

• in your home, run mkdir mount
• mkdir vm_001_ubuntu or whatever
• mount it sshfs root@127.0.0.1:/ -p 20022 ./vm_001_ubuntu
• verify with findmnt

enjoy

TLDR qemu

buat file namanya setup-network.sh, lalu isi dengan

sudo ip link add br0-lan type bridge
sudo ip tuntap add tap0 mode tap
sudo ip tuntap add tap1 mode tap
sudo ip tuntap add tap2 mode tap

sudo ip link set tap0 master br0-lan
sudo ip link set tap1 master br0-lan
sudo ip link set tap2 master br0-lan

sudo ip link set br0-lan up
sudo ip link set tap0 up
sudo ip link set tap1 up
sudo ip link set tap2 up

sudo ip a

lalu lakukan ini agar bisa di exec via terminal

• sudo chmod 666 setup-network.sh
• sh ./setup-network.sh

running qemu

run routerOS

# download dahulu chr-7.19.1.img dari web resminya
qemu-img convert -f raw -O qcow2 chr-7.19.1.img chr7.qcow2

# run chr
# port yang dipakai
# - 8291: untuk winbox
# - 30022: untuk keperluan sftp, utak atik file hotspot nya
qemu-system-x86_64 \
	-enable-kvm \
	-smp 4 \
	-m 256M \
	-drive file=chr7.qcow2,format=qcow2 \
	-boot order=d \
	-net user,hostfwd=tcp::8291-:8291,hostfwd=tcp::30022-:22 \
	-net nic \
	-nographic \
	-netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
	-device virtio-net-pci,netdev=net0,mac=02:11:2a:3b:ff:c3

ubuntu server

buat image (HDD virtual untuk os kita nanti)

qemu-img create -f qcow2 ubuntu-server.img 15G

run iso (saat install)

qemu-system-x86_64 \
	-enable-kvm \
	-boot order=d \
	-cdrom GANTI_NAMA_ISO_UBUNTU.iso \
	-drive file=ubuntu-server.img,format=qcow2 \
	-m 4G \
	-smp 4 \
	-netdev user,id=net0,hostfwd=tcp::20022-:22,hostfwd=tcp::10000-:5432 \
	-device e1000,netdev=net0 \
	-netdev tap,id=net1,ifname=tap1,script=no,downscript=no \
	-device virtio-net-pci,netdev=net1,mac=02:11:2a:3b:aa:c4 \
	-vga virtio

lalu run hasil image (Default no graphic, jadi ntar kita ssh pakai ssh -p 20022 root@127.0.0.1 biar bisa copas)

dan juga kita optimalisasi dengan cara pakai nographic, tanpa cdrom, map port yang dibutuhkan saja

port yg di forward:

• 20022: untuk remote
• 10302: untuk keperluan ngoding laravel
• 10000: untuk keperluan postgresql, siapa tahu mau di remote via adminer, pgadmin dari local, atau cuman psql dll

qemu-system-x86_64 \
	-enable-kvm \
	-boot order=d \
	-drive file=ubuntu-server.img,format=qcow2 \
	-m 4G \
	-smp 4 \
	-netdev user,id=net0,hostfwd=tcp::20022-:22,hostfwd=tcp::10302-:10302,hostfwd=tcp::10000-:5432 \
	-device e1000,netdev=net0 \
	-netdev tap,id=net1,ifname=tap1,script=no,downscript=no \
	-device virtio-net-pci,netdev=net1,mac=02:11:2a:3b:aa:c4 \
	-vga virtio \
	-nographic

VPS KVM siap jadi bahan ujicoba, langsung test saja ke ssh -p 20022 root@127.0.0.1

experimental (jangan dipakai / do not use)

untuk catatan pribadi bahan experimental UEFI, KVM, sama custom machine q35

experimental 1 (safest)

qemu-system-x86_64 \
    -name guest=ubuntu22.04,debug-threads=on \
    -machine type=pc,accel=kvm \
    -cpu host \
    -enable-kvm \
    -boot order=d \
    -drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd \
    -drive if=pflash,format=raw,file=OVMF_VARS_ubuntu_server_gpt.4m.fd \
    -drive file=ubuntu-server.img,format=qcow2 \
    -m 4G \
    -smp 4 \
    -netdev user,id=net0,hostfwd=tcp::20022-:22,hostfwd=tcp::10000-:5432 \
    -device virtio-net-pci,netdev=net0 \
    -netdev tap,id=net1,ifname=tap1,script=no,downscript=no \
    -device virtio-net-pci,netdev=net1,mac=02:11:2a:3b:aa:c4

dgn iso

qemu-system-x86_64 \
	-name guest=ubuntu22.04,debug-threads=on \
	-machine type=pc,accel=kvm \
	-cpu host \
	-enable-kvm \
	-boot order=d \
	-cdrom a.iso \
	-drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd \
	-drive if=pflash,format=raw,file=OVMF_VARS_ubuntu_server_gpt.4m.fd \
	-drive file=ubuntu-server.img,format=qcow2 \
	-m 4G \
	-smp 4 \
	-netdev user,id=net0,hostfwd=tcp::20022-:22,hostfwd=tcp::10302-:10302,hostfwd=tcp::10000-:5432 \
	-device e1000,netdev=net0 \
	-netdev tap,id=net1,ifname=tap1,script=no,downscript=no \
	-device virtio-net-pci,netdev=net1,mac=02:11:2a:3b:aa:c4 \
	-vga virtio

custom machine chipset PC, KVM, UEFI headless

qemu-system-x86_64 \
	-name guest=ubuntu22.04,debug-threads=on \
	-machine type=pc,accel=kvm \
	-cpu host \
	-enable-kvm \
	-boot order=d \
	-drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd \
	-drive if=pflash,format=raw,file=OVMF_VARS_ubuntu_server_gpt.4m.fd \
	-drive file=ubuntu-server.img,format=qcow2 \
	-m 4G \
	-smp 4 \
	-netdev user,id=net0,hostfwd=tcp::20022-:22,hostfwd=tcp::10302-:10302,hostfwd=tcp::10000-:5432 \
	-device e1000,netdev=net0 \
	-netdev tap,id=net1,ifname=tap1,script=no,downscript=no \
	-device virtio-net-pci,netdev=net1,mac=02:11:2a:3b:aa:c4 \
	-vga virtio \
	-nographic

working (part 2)

qemu-system-x86_64 \
          -enable-kvm \
          -boot order=d \
          -drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd \
          -drive if=pflash,format=raw,file=OVMF_VARS_ubuntu_server_gpt.4m.fd \
          -drive file=ubuntu-server.img,format=qcow2 \
          -m 4G \
          -smp 4 \
          -netdev user,id=net0,hostfwd=tcp::20022-:22,hostfwd=tcp::10000-:5432 \
          -device e1000,netdev=net0 \
          -netdev tap,id=net1,ifname=tap1,script=no,downscript=no \
          -device virtio-net-pci,netdev=net1,mac=02:11:2a:3b:aa:c4 \
          -vga virtio

no iso (headless)

qemu-system-x86_64 \
	-name guest=ubuntu22.04,debug-threads=on \
	-machine type=q35,accel=kvm \
	-cpu host \
	-enable-kvm \
	-boot order=d \
	-drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd \
	-drive if=pflash,format=raw,file=OVMF_VARS_ubuntu_server_gpt.4m.fd \
	-drive file=ubuntu-server.img,format=qcow2 \
	-m 4G \
	-smp 4 \
	-netdev user,id=net0,hostfwd=tcp::20022-:22,hostfwd=tcp::10000-:5432 \
	-device e1000,netdev=net0 \
	-netdev tap,id=net1,ifname=tap1,script=no,downscript=no \
	-device virtio-net-pci,netdev=net1,mac=02:11:2a:3b:aa:c4

qemu q35 vs i440fx

q35

this is lspci output from ubuntu server

i440fx

http://wiki.qemu.org/images/4/4e/Q35.pdf

http://wiki.qemu.org/images/f/f6/PCIvsPCIe.pdf

http://git.qemu.org/?p=qemu.git;a=blob;f=docs/pcie.txt

qemu & dedicated nic + macvtap

docs ini memahas solusi agar bisa akses tap (dari vm) tetapi hw yang diakses adalah nic asli, diusahakan tanpa bridge dan NAT NAT

ADA 2 strategi

• add iface fisik ke bridge, lalu attach bridge tersebut ke vm.
• pakai macvtap, dengan cara meredirect packet yang datang, ke arah tap device menggunakan fd (file descriptor). kita akan (ab)use option -net tap di qemu

strategi lain (tidak direkomendasikan, diluar topik macvtap)

• membuat virtual eth pair, redirect pakai iptables
• pakai -netdev user (somewhat slow, need triage)

normal

bagian ini tanpa macvtap ataupun hal hal lain dahulu, pure qemu bridging

configuration

• eth1: mostly WAN
• eth2: tap

qemu-system-x86_64 \
    -enable-kvm \
    -smp 4 -m 256M \
    -drive file=chr7.qcow2,format=qcow2 \
    -boot order=d \
    -net user,hostfwd=tcp::8291-:8291,hostfwd=tcp::10022-:22 \
    -net nic \
    -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
    -device virtio-net-pci,netdev=net0,mac=02:11:2a:3b:ff:c3 \
    -nographic

qemu-system-x86_64 \
	-name guest=ubuntu22.04 \
	-machine type=pc,accel=kvm \
	-cpu host -m 4G -smp 4 \
	-enable-kvm \
	-boot order=d \
	-drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd \
	-drive if=pflash,format=raw,file=OVMF_VARS_ubuntu_server_gpt.4m.fd \
	-drive file=ubuntu-server.img,format=qcow2 \
	-netdev user,id=net0,hostfwd=tcp::20022-:22,hostfwd=tcp::10000-:5432,hostfwd=tcp::10302-:10302 \
	-device virtio-net-pci,netdev=net0 \
	-netdev tap,id=net1,ifname=tap1,script=no,downscript=no \
	-device virtio-net-pci,netdev=net1,mac=02:11:2a:3b:aa:c4 \
	-nographic

network stack note:

• ens3: come from qemu bridge (for ssh purpose)
• ens4: come from tap1 (connected to bridge, internal lan)

netplan configuration

network:
  version: 2
  ethernets:
    ens4:
      addresses:
        - 192.168.1.2/24
      routes:
        - to: default
          via: 192.168.1.1
    ens3:
      dhcp4: true

// ip a 

2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic ens3
       valid_lft 86314sec preferred_lft 86314sec
    inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute 
       valid_lft 86317sec preferred_lft 14317sec
    inet6 fe80::5054:ff:fe12:3456/64 scope link 
       valid_lft forever preferred_lft forever
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:11:2a:3b:aa:c4 brd ff:ff:ff:ff:ff:ff
    altname enp0s4
    inet 192.168.1.2/24 brd 192.168.1.255 scope global ens4
       valid_lft forever preferred_lft forever
    inet6 fe80::11:2aff:fe3b:aac4/64 scope link 
       valid_lft forever preferred_lft forever

alpine install SSH + firefox lite qemu

create image

qemu-img create -f qcow2 alpine-client.qcow2 2G

RUN

THIS is dry run

qemu-system-x86_64 \
	-enable-kvm \
	-m 1G -smp 4 -cpu host \
	-boot d \
	-drive file=alpine-client.qcow2,format=qcow2 \
	-cdrom ./iso/alpine-virt-3.21.3-x86_64.iso \
	-netdev user,id=net0,hostfwd=tcp::30022-:22 \
	-device virtio-net-pci,netdev=net0
	-vga virtio

Database

psql command

General

• \bind [PARAM]... set query parameters
• \copyright show PostgreSQL usage and distribution terms
• \crosstabview [COLUMNS] execute query and display result in crosstab
• \errverbose show most recent error message at maximum verbosity
• \g [(OPTIONS)] [FILE] execute query (and send result to file or |pipe); \g with no arguments is equivalent to a semicolon
• \gdesc describe result of query, without executing it
• \gexec execute query, then execute each value in its result
• \gset [PREFIX] execute query and store result in psql variables
• \gx [(OPTIONS)] [FILE] as \g, but forces expanded output mode
• \q quit psql
• \watch [[i=]SEC] [c=N] [m=MIN] execute query every SEC seconds, up to N times, stop if less than MIN rows are returned

Help

• \? [commands] show help on backslash commands
• \? options show help on psql command-line options
• \? variables show help on special variables
• \h [NAME] help on syntax of SQL commands, * for all commands

Query Buffer

• \e [FILE] [LINE] edit the query buffer (or file) with external editor
• \ef [FUNCNAME [LINE]] edit function definition with external editor
• \ev [VIEWNAME [LINE]] edit view definition with external editor
• \p show the contents of the query buffer
• \r reset (clear) the query buffer
• \s [FILE] display history or save it to file
• \w FILE write query buffer to file

Input/Output

• \copy ... perform SQL COPY with data stream to the client host
• \echo [-n] [STRING] write string to standard output (-n for no newline)
• \i FILE execute commands from file
• \ir FILE as \i, but relative to location of current script
• \o [FILE] send all query results to file or |pipe
• \qecho [-n] [STRING] write string to \o output stream (-n for no newline)
• \warn [-n] [STRING] write string to standard error (-n for no newline)

Conditional

• \if EXPR begin conditional block
• \elif EXPR alternative within current conditional block
• \else final alternative within current conditional block
• \endif end conditional block

Informational

(options: S = show system objects, + = additional detail)

• \d[S+] list tables, views, and sequences
• \d[S+] NAME describe table, view, sequence, or index
• \da[S] [PATTERN] list aggregates
• \dA[+] [PATTERN] list access methods
• \dAc[+] [AMPTRN [TYPEPTRN]] list operator classes
• \dAf[+] [AMPTRN [TYPEPTRN]] list operator families
• \dAo[+] [AMPTRN [OPFPTRN]] list operators of operator families
• \dAp[+] [AMPTRN [OPFPTRN]] list support functions of operator families
• \db[+] [PATTERN] list tablespaces
• \dc[S+] [PATTERN] list conversions
• \dconfig[+] [PATTERN] list configuration parameters
• \dC[+] [PATTERN] list casts
• \dd[S] [PATTERN] show object descriptions not displayed elsewhere
• \dD[S+] [PATTERN] list domains
• \ddp [PATTERN] list default privileges
• \dE[S+] [PATTERN] list foreign tables
• \des[+] [PATTERN] list foreign servers
• \det[+] [PATTERN] list foreign tables
• \deu[+] [PATTERN] list user mappings
• \dew[+] [PATTERN] list foreign-data wrappers
• \df[anptw][S+] [FUNCPTRN [TYPEPTRN ...]] list [only agg/normal/procedure/trigger/window] functions
• \dF[+] [PATTERN] list text search configurations
• \dFd[+] [PATTERN] list text search dictionaries
• \dFp[+] [PATTERN] list text search parsers
• \dFt[+] [PATTERN] list text search templates
• \dg[S+] [PATTERN] list roles
• \di[S+] [PATTERN] list indexes
• \dl[+] list large objects, same as \lo_list
• \dL[S+] [PATTERN] list procedural languages
• \dm[S+] [PATTERN] list materialized views
• \dn[S+] [PATTERN] list schemas
• \do[S+] [OPPTRN [TYPEPTRN [TYPEPTRN]]] list operators
• \dO[S+] [PATTERN] list collations
• \dp[S] [PATTERN] list table, view, and sequence access privileges
• \dP[itn+] [PATTERN] list [only index/table] partitioned relations [n=nested]
• \drds [ROLEPTRN [DBPTRN]] list per-database role settings
• \drg[S] [PATTERN] list role grants
• \dRp[+] [PATTERN] list replication publications
• \dRs[+] [PATTERN] list replication subscriptions
• \ds[S+] [PATTERN] list sequences
• \dt[S+] [PATTERN] list tables
• \dT[S+] [PATTERN] list data types
• \du[S+] [PATTERN] list roles
• \dv[S+] [PATTERN] list views
• \dx[+] [PATTERN] list extensions
• \dX [PATTERN] list extended statistics
• \dy[+] [PATTERN] list event triggers
• \l[+] [PATTERN] list databases
• \sf[+] FUNCNAME show a function's definition
• \sv[+] VIEWNAME show a view's definition
• \z[S] [PATTERN] same as \dp

Large Objects

• \lo_export LOBOID FILE write large object to file
• \lo_import FILE [COMMENT] read large object from file
• \lo_list[+] list large objects
• \lo_unlink LOBOID delete a large object

Formatting

• \a toggle between unaligned and aligned output mode
• \C [STRING] set table title, or unset if none
• \f [STRING] show or set field separator for unaligned query output
• \H toggle HTML output mode (currently off)
• \pset [NAME [VALUE]] set table output option (border|columns|csv_fieldsep|expanded|fieldsep| fieldsep_zero|footer|format|linestyle|null| numericlocale|pager|pager_min_lines|recordsep| recordsep_zero|tableattr|title|tuples_only| unicode_border_linestyle|unicode_column_linestyle| unicode_header_linestyle|xheader_width)
• \t [on|off] show only rows (currently off)
• \T [STRING] set HTML tag attributes, or unset if none
  \x [on|off|auto] toggle expanded output (currently off)

  Connection

  • \c[onnect] {[DBNAME|- USER|- HOST|- PORT|-] | conninfo} connect to new database (currently "fadev_infra")
  • \conninfo display information about current connection
  • \encoding [ENCODING] show or set client encoding
  • \password [USERNAME] securely change the password for a user

  Operating System

  • \cd [DIR] change the current working directory
  • \getenv PSQLVAR ENVVAR fetch environment variable
  • \setenv NAME [VALUE] set or unset environment variable
  • \timing [on|off] toggle timing of commands (currently off)
  • ! [COMMAND] execute command in shell or start interactive shell

  Variables

  • \prompt [TEXT] NAME prompt user to set internal variable
  • \set [NAME [VALUE]] set internal variable, or list all if no parameters
  • \unset NAME unset (delete) internal variable

  psql snippet

  psql

  • \df *.* Lists all relations
  • \l Lists all database

  command

  • DROP TABLE table_name CASCADE; used when you want to remove all object depend on them.

  PostgreSQL database creation flow

  in order to create database (for safety, I use postgres account). I use this step.

  first, we need to login as postgres user, use this sudo -u postgres psql, then do

  CREATE DATABASE EXAMPLE_DB;
  

  this is optional

  CREATE USER EXAMPLE_USER WITH ENCRYPTED PASSWORD 'Sup3rS3cret';
  

  grant

  GRANT ALL PRIVILEGES ON DATABASE EXAMPLE_DB TO EXAMPLE_USER;
  

  now, change the state using \c EXAMPLE_DB postgres, execute this command

  GRANT ALL ON SCHEMA public TO EXAMPLE_USER;
  

  then, change ownership

  ALTER DATABASE my_database OWNER TO my_database_user;
  

  notes: https://stackoverflow.com/questions/67276391/why-am-i-getting-a-permission-denied-error-for-schema-public-on-pgadmin-4

  reverse

  first, we change to postgres account

  su postgres
  

  list database: \l

  remove:

  DROP DATABASE abc;
  

  drop user, first list users first: \du+

  drop:

  DROP USER IF EXISTS foobar;
  

  revoking access:

  REVOKE ALL PRIVILEGES ON SCHEMA public FROM foobar;
  
  -- for tables, seq, and func
  REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM foobar;
  REVOKE ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public FROM foobar;
  REVOKE ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA public FROM foobar;
  

  check object ownership

  SELECT 
      n.nspname AS schema,
      c.relname AS object,
      c.relkind AS type
  FROM 
      pg_class c
  JOIN 
      pg_roles r ON c.relowner = r.oid
  JOIN 
      pg_namespace n ON n.oid = c.relnamespace
  WHERE 
      r.rolname = 'foobar';
  

  or \ddp -- in psql

  

  granting

  granting all sequences

  GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO radius;
  

  how to use SQLX cli tools for db migration such laravel migrations

  let's meet with this crate, https://crates.io/crates/sqlx-cli, this tools allow you to use db migration such as laravel migration tools;

  first, let's create env variable

  export DATABASE_URL=postgres://fadhil_riyanto:<your_pass>@localhost/<target_db>
  

  then, the things that we do next is only

  sqlx migrate add -r <name>
  

  running

  sqlx migrate run
  

  Postgresql internal (server)

  this thing explain the internal of pgsql system (not the source code I mean)

  difference between catalog vs schema

  key point:

  • schema is folder like, grouping table together, can be created using

    CREATE SCHEMA my_catalog;
    CREATE TABLE my_catalog.metadata_table (...);
    

  • catalog is system schema, can't be created manually, part of internal pgsql system

  • view is a resulted table from XYZ queries, example

    CREATE VIEW active_users AS 
    	SELECT id, name, 
    	FROM
    		users 
    	WHERE
    		active = true;
    

    then query it

    SELECT * FROM active_users;
    

  pg_catalog inside

  this is some stuff inside of pg catalog, you can do that using \dt pg_catalog.*

  • pg_aggregate: Stores information about aggregate functions (like SUM, AVG, etc.).
  • pg_am: Lists access methods for indexes (e.g., btree, hash).
  • pg_amop: Defines operators used in access methods.
  • pg_amproc: Defines support functions used in access methods.
  • pg_attrdef: Stores default values for columns.
  • pg_attribute: Contains column definitions for all tables.
  • pg_auth_members: Shows role memberships (who is a member of what).
  • pg_authid: Stores user/role definitions (superuser access required).
  • pg_cast: Contains rules for casting between data types.
  • pg_class: Contains all table-like objects (tables, views, indexes, sequences, etc.).
  • pg_collation: Defines collations (rules for string comparison).
  • pg_constraint: Stores constraints like PRIMARY KEY, UNIQUE, CHECK, and FOREIGN KEY.
  • pg_conversion: Defines character set conversions.
  • pg_database: Stores information about each database in the cluster.
  • pg_db_role_setting: Stores per-user/per-database configuration settings (GUCs).
  • pg_default_acl: Defines default privileges for newly created objects.
  • pg_depend: Stores dependency relationships between database objects.
  • pg_description: Stores comments/descriptions on database objects.
  • pg_enum: Stores values for ENUM data types.
  • pg_event_trigger: Stores event trigger definitions (triggers on DDL commands).
  • pg_extension: Tracks installed extensions (like uuid-ossp, pgcrypto, etc.).
  • pg_foreign_data_wrapper: Stores definitions of foreign data wrappers (FDW).
  • pg_foreign_server: Stores foreign servers used by FDWs.
  • pg_foreign_table: Stores metadata for foreign tables.
  • pg_index: Contains metadata about indexes (e.g., indexed columns).
  • pg_inherits: Stores table inheritance relationships.
  • pg_init_privs: Records original privileges on built-in objects.
  • pg_language: Stores information about supported procedural languages.
  • pg_largeobject: Stores the actual data of large objects (blobs).
  • pg_largeobject_metadata: Stores metadata about large objects.
  • pg_namespace: Lists all schemas in the database (IMPORTANT)
  • pg_opclass: Stores index operator classes (how a datatype is indexed).
  • pg_operator: Stores SQL operators (like =, <, +, etc.).
  • pg_opfamily: Groups related operator classes.
  • pg_parameter_acl: Stores access control for configuration parameters (PostgreSQL 16+).
  • pg_partitioned_table: Stores metadata for partitioned tables.
  • pg_policy: Stores row-level security policies.
  • pg_proc: Contains all function and procedure definitions.
  • pg_publication: Stores logical replication publications.
  • pg_publication_namespace: Links publications to schemas.
  • pg_publication_rel: Links publications to individual tables.
  • pg_range: Stores definitions of range types (e.g., int4range).
  • pg_replication_origin: Tracks origins for logical replication.
  • pg_rewrite: Stores query rewrite rules (used in views, rules).
  • pg_seclabel: Stores security labels for database objects.
  • pg_sequence: Contains metadata for sequence generators.
  • pg_shdepend: Tracks dependencies involving shared objects (like roles, databases).
  • pg_shdescription: Stores comments on shared objects.
  • pg_shseclabel: Stores security labels on shared objects.
  • pg_statistic: Stores planner statistics for columns.
  • pg_statistic_ext: Stores extended planner statistics (multi-column, NDV, etc.).
  • pg_statistic_ext_data: Contains actual values for extended statistics.
  • pg_subscription: Defines logical replication subscriptions.
  • pg_subscription_rel: Lists tables included in subscriptions.
  • pg_tablespace: Lists all tablespaces (disk locations for data).
  • pg_transform: Stores type transformation functions for procedural languages.
  • pg_trigger: Stores triggers on tables.
  • pg_ts_config: Stores full-text search configurations.
  • pg_ts_config_map: Maps text search config tokens to dictionaries.
  • pg_ts_dict: Stores text search dictionaries.
  • pg_ts_parser: Defines tokenizers for full-text search.
  • pg_ts_template: Defines templates for building text search dictionaries.
  • pg_type: Stores all data types (built-in, custom, enum, composite). (IMPORTANT)
  • pg_user_mapping: Maps users to foreign servers.

  pg_catalog.pg_database details

  this docs can be found in https://www.postgresql.org/docs/16/catalog-pg-database.html

  Column Name	Type	Description
  datname	name	The name of the database.
  datdba	oid	The OID of the role (user) that owns the database. Use pg_get_userbyid(datdba) to resolve it.
  encoding	int	The character encoding of the database (e.g., UTF8 = 6). Use pg_encoding_to_char(encoding) to get the name.
  datlocprovider	char	Locale provider used (c = libc, i = ICU).
  datistemplate	bool	If true, the database can be used as a template for CREATE DATABASE ... TEMPLATE.
  datallowconn	bool	If false, connections to this database are not allowed (except by superusers).
  datconnlimit	int	The maximum number of concurrent connections allowed (-1 = no limit).
  datlastsysoid	oid	The last system OID used in this database at creation (mainly historical).
  datfrozenxid	xid	The transaction ID at which all tuples are known to be frozen (related to VACUUM).
  datminmxid	xid	The minimum multixact ID that is still considered potentially unfrozen.
  dattablespace	oid	OID of the default tablespace for the database. Use pg_tablespace to resolve.
  datcollate	name	LC_COLLATE setting (how strings are sorted).
  datctype	name	LC_CTYPE setting (how character classification works).
  daticulocale	text	ICU locale (used if datlocprovider = 'i').
  datcollversion	text	Version of the collation used (important for collation versioning with ICU).
  datacl	aclitem[]	Access privileges (GRANTs), stored as an array of ACL items.