Peter Pap's Technowiki - New pages [en]

Rancher Login Page Blank

2025-06-27T03:09:16Z

Ppapa:

So, the Rancher login page is loading blank. You log into one of the Rancher nodes and see this:

# kubectl get node
E0627 01:17:45.898377 2181261 memcache.go:265] couldn't get current server API group list: an error on the server ("dial tcp 10.43.0.1:443: connect: connection refused") has prevented the request from succeeding
E0627 01:17:45.899229 2181261 memcache.go:265] couldn't get current server API group list: an error on the server ("dial tcp 10.43.0.1:443: connect: connection refused") has prevented the request from succeeding
E0627 01:17:45.901039 2181261 memcache.go:265] couldn't get current server API group list: an error on the server ("dial tcp 10.43.0.1:443: connect: connection refused") has prevented the request from succeeding
E0627 01:17:45.901660 2181261 memcache.go:265] couldn't get current server API group list: an error on the server ("dial tcp 10.43.0.1:443: connect: connection refused") has prevented the request from succeeding
E0627 01:17:45.903403 2181261 memcache.go:265] couldn't get current server API group list: an error on the server ("dial tcp 10.43.0.1:443: connect: connection refused") has prevented the request from succeeding
Error from server (InternalError): an error on the server ("dial tcp 10.43.0.1:443: connect: connection refused") has prevented the request from succeeding

You run the following command:

# openssl s_client -connect 127.0.0.1:443 -showcerts </dev/null | openssl x509 -noout -text
Can't use SSL_get_servername
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=18:self-signed certificate
verify return:1
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=10:certificate has expired
notAfter=Jun 26 01:04:33 2025 GMT
verify return:1
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
notAfter=Jun 26 01:04:33 2025 GMT
verify return:1
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
82:a2:22:9a:7f:ff:06:e3:cd:4e:1f:23:a6:16:fc:05
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
Validity
Not Before: Jun 26 01:04:33 2024 GMT
'''Not After : Jun 26 01:04:33 2025 GMT'''

Look for the "'''Not After'''" field. If it's in the past, then your "'''Kubernetes Ingress Controller Fake Certificate'''" has expired! This certificate is used for inter Rancher node communication. To make it renew, reboot the node. Wait for it to come back up, then reboot the next node etc. Rancher should be back.

Rancher

2025-06-27T03:04:42Z

Ppapa: Created page with "<ul> <li>Rancher Login Page Blank</li> </UL>"

<ul>
<li>[[Rancher Login Page Blank]]</li>
</UL>

Kubernetes

2025-06-27T03:03:45Z

Ppapa: Created page with "<ul> <li>Rancher</li> </UL>"

<ul>
<li>[[Rancher]]</li>
</UL>

Verify that a private key matches a certificate

2024-02-12T04:41:41Z

Ppapa:

To verify that an RSA private key matches the RSA public key in a certificate you need to i) verify the consistency of the private key and ii) compare the modulus of the public key in the certificate against the modulus of the private key.

To verify the consistency of the RSA private key and to view its modulus:
openssl rsa -modulus -noout -in ''my.key'' | openssl md5

openssl rsa -check -noout -in ''my.key''
RSA Key is ok
If it doesn't say 'RSA key ok', it isn't OK!"

To view the modulus of the RSA public key in a certificate:
openssl x509 -modulus -noout -in ''my.crt'' | openssl md5

If the first commands shows any errors, or if the modulus of the public key in the certificate and the modulus of the private key do not exactly match, then you're not using the correct private key. You can either create a brand new key and CSR and contact support, or you can do a search for any other private keys on the system and see if they match.

OpenSSL

2024-02-12T04:40:20Z

Ppapa:

<ul>
<li>[[Verify that a private key matches a certificate]]</li>
</ul>

Show files for a given tablespace

2021-03-11T00:43:25Z

Ppapa:

set linesize 160
set wrap off
column name format a20
column filename format a50

select ts.name, df.name as FILENAME
from V_$TABLESPACE ts, V_$DATAFILE df
where df.TS# = ts.TS#
order by ts.name , df.name;

Email alerts failing to send from Dell iDRAC v9

2020-10-08T05:35:01Z

Ppapa:

If you have a Dell server with an iDRAC9, release 4.x, you'll no doubt no longer be receiving email from it. The reason for this is:

''After iDRAC is upgraded to version 4.00.00.00, you may stop receiving encrypted email alerts from iDRAC, if the external email server does not support encryption. iDRAC firmware version 4.00.00.00 introduces a user-selectable encryption option and the default protocol is StartTLS.''

The tell-tale sign of this is when you see the following message in your mail server logs:

Dec 10 00:33:28 mailhost01 sendmail[22446]: 1BA0XSsj022446: idrac.mydomain.com [192.168.2.2] did not issue MAIL/EXPN/VRFY/ETRN during connection to Daemon0

To start receiving email messages again, disable the email encryption by using the following RACADM command:

racadm set idrac.RemoteHosts.ConnectionEncryption None

As of iDRAC9 release 5.x, you can change this from the GUI:

- iDRAC Settings
-> Settings
- SMTP (Email) Server Settings
-> Connection Encryption : '''None'''

Dell

2020-10-08T05:31:29Z

Ppapa:

<ul>
<li>[[Email alerts failing to send from Dell iDRAC v9]]</li>
</UL>

Changing sendmail connect domain macro, Dj, in sendmail.mc

2020-10-02T01:17:59Z

Ppapa:

Generally, hacking your sendmail.cf is a very bad idea! Any changes you want to make should be done in the sendmail.mc file and then the sendmail.cf file built from it. This example is where your mail relay has an internal hostname (eg. internal01.mail.servers.net) but, when it connects to the outside world, you want it to use a different name, eg. relay.myorg.com. Often this is where you have multiple internal mail servers, but they share a common external IP. To change this setting:

1. Edit sendmail.mc and add/change this line:

define(`confDOMAIN_NAME', `relay.myorg.com')dnl

Obviously, change the value to whatever it is that you need for your org.

2. Rebuild sendmail.cf for your platform. It's different on Solaris and Linux, so do what works for you.

3. Restart sendmail.

Increasing disk space on a VM using Logical Volumes

2020-10-01T01:28:57Z

Ppapa:

So you have a CentOS VM that uses logical volumes and it's running out of space on a given filesystem. You can increase the disk space as described [https://kb.vmware.com/s/article/1006371 here]. Slight adjustments below:

1. Power off the virtual machine.

2. Edit the virtual machine settings and extend the virtual disk size.

3. Power on the virtual machine.

4. Identify the device name, which is by default /dev/sda, and confirm the new size by running the command:

# fdisk -l

5. Create a new primary partition:

# fdisk /dev/sda (depending the results of the step 4)

Press '''p''' to print the partition table to identify the number of partitions. By default, there are 2: sda1 and sda2.

Press '''n''' to create a new primary partition.

Press '''p''' for primary.

Press '''3''' for the partition number, depending on the output of the partition table print.

Press Enter two times.

Press '''t''' to change the system's partition ID.

Press '''3''' to select the newly creation partition.

Type '''8e''' to change the Hex Code of the partition for Linux LVM.

Press '''w''' to write the changes to the partition table.

6. Restart the virtual machine.

7. Run this command to convert the new partition to a physical volume:

'''Note:''' The number for the sda can change depending on system setup. Use the sda number that was created in step 5.

# pvcreate /dev/sda3

8. Run this command to extend the physical volume:

# vgextend VolGroup00 /dev/sda3

'''Note:''' To determine which volume group to extend, use the command vgdisplay.

9. Run this command to verify how many physical extents are available to the Volume Group:

# vgdisplay VolGroup00 | grep "Free"

10. Run the following command to extend the Logical Volume:

# lvextend -L+#G /dev/VolGroup00/LogVol00

Where # is the number of Free space in GB available as per the previous command. Use the full number output from Step 10 including any decimals.

'''Note:''' To determine which logical volume to extend, use the command lvdisplay.

11. Run the following command to expand the ext3 filesystem online, inside of the Logical Volume:

# resize2fs /dev/VolGroup00/LogVol00

Configure NetBackup client based encryption

2019-04-29T00:14:20Z

Ppapa: Created page with "Documented quite well [https://www.veritas.com/support/en_US/article.100021401 here]"

Documented quite well [https://www.veritas.com/support/en_US/article.100021401 here]

Veritas

2019-04-29T00:09:57Z

Ppapa:

<ul>
<li>[[Installing NetBackup on Linux]]</li>
<li>[[Manually expiring images from disk in NetBackup]]</li>
<li>[[Netbackup – Increase Number of Days in Activity Monitor Job History]]</li>
<li>[[Overriding the Symantec IPS on a NetBackup Appliance]]</li>
<li>[[Configure NetBackup client based encryption]]</li>
<li>[[All compatible drive paths are down but media is available]]</li>
</UL>

Replacing a failed disk in a ZFS/ZPOOL raid array

2019-02-27T01:20:47Z

Ppapa:

This has been used on a SPARC S7-2, but should be relevant to most modern sparc based servers:

1. Identify the failed disk in the zpool array:

# '''zpool status'''
pool: rpool
state: DEGRADED

config:

NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
c0t5000CCA02F613ACCd0 ONLINE 0 0 0
'''c0t5000CCA02D0F6C44d0''' UNAVAIL 0 0 0

2. Find the device path for the drive you want to remove:

# '''diskinfo D:devchassis-path'''
D:devchassis-path c:occupant-compdev
---------------------------------- ---------------------
/dev/chassis/SYS/HDD0/disk c0t5000CCA02F613ACCd0
/dev/chassis/SYS/HDD1/disk '''c0t5000CCA02D0F6C44d0'''
/dev/chassis/SYS/HDD2 -
/dev/chassis/SYS/HDD3 -
/dev/chassis/SYS/HDD4 -
/dev/chassis/SYS/HDD5 -
/dev/chassis/SYS/HDD6 -
/dev/chassis/SYS/HDD7 -
/dev/chassis/SYS/MB/EUSB_DISK/disk c1t0d0

3. Check the drive's status:

# '''cfgadm -al'''
Ap_Id Type Receptacle Occupant Condition
/SYS/DBP/NVME0 unknown empty unconfigured unknown
/SYS/DBP/NVME1 unknown empty unconfigured unknown
/SYS/DBP/NVME2 unknown empty unconfigured unknown
/SYS/DBP/NVME3 unknown empty unconfigured unknown
c3 scsi-sas connected configured unknown
c3::w5000cca02f613acd,0 disk-path connected configured unknown
c4 scsi-sas connected configured unknown
'''c4::w5000CCA02D0F6C45,0 disk-path connected configured unknown'''
usb0/1 usb-storage connected configured ok
usb0/2 usb-hub connected configured ok

'''NOTICE''' the slight mismatch between the output of the last two commands.

4. Unconfigure the drive:

# '''cfgadm -c unconfigure c4::w5000CCA02D0F6C45,0'''

and check that it worked:

# '''cfgadm -al'''
Ap_Id Type Receptacle Occupant Condition
/SYS/DBP/NVME0 unknown empty unconfigured unknown
/SYS/DBP/NVME1 unknown empty unconfigured unknown
/SYS/DBP/NVME2 unknown empty unconfigured unknown
/SYS/DBP/NVME3 unknown empty unconfigured unknown
c3 scsi-sas connected configured unknown
c3::w5000cca02f613acd,0 disk-path connected configured unknown
c4 scsi-sas connected configured unknown
c4::w5000CCA02D0F6C45,0 disk-path connected '''unconfigured''' unknown

5. Turn on the Ok to Remove indicator for that drive:

# '''fmadm set-indicator /dev/chassis/SYS/HDD1/disk ok2rm on'''

and check that it worked:

# '''fmadm get-indicator /dev/chassis/SYS/HDD1/disk ok2rm'''
The indicator (ok2rm) is set to on.

6. Remove the failed drive and replace with the new one.

7. The new drive should be configured automatically, but check it anyway:

# '''cfgadm -al'''
Ap_Id Type Receptacle Occupant Condition
/SYS/DBP/NVME0 unknown empty unconfigured unknown
/SYS/DBP/NVME1 unknown empty unconfigured unknown
/SYS/DBP/NVME2 unknown empty unconfigured unknown
/SYS/DBP/NVME3 unknown empty unconfigured unknown
c3 scsi-sas connected configured unknown
c3::w5000cca02f613acd,0 disk-path connected configured unknown
c4 scsi-sas connected configured unknown
c4::w5000cca07d293ca1,0 disk-path connected '''configured''' unknown
usb0/1 usb-storage connected configured ok
usb0/2 usb-hub connected configured ok

If not, configure with:

# '''cfgadm -c unconfigure c4::w5000cca07d293ca1,0'''

8. Get the name od the new drive from the format command:

# '''format'''
Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c0t5000CCA02F613ACCd0 <HGST-H101860SFSUN600G-A990-558.91GB>
/scsi_vhci/disk@g5000cca02f613acc
/dev/chassis/SYS/HDD0/disk
1. '''c0t5000CCA07D293CA0d0''' <HGST-H101860SFSUN600G-A990-558.91GB>
/scsi_vhci/disk@g5000cca07d293ca0
/dev/chassis/SYS/HDD1/disk
2. c1t0d0 <VT-eUSB-7722-1.91GB>
/pci@300/pci@1/pci@0/pci@2/usb@0/storage@1/disk@0,0
/dev/chassis/SYS/MB/EUSB_DISK/disk
Specify disk (enter its number):

9. Replace the failed disk with the new disk in the zpool:

# '''zpool replace rpool c0t5000CCA02D0F6C44d0 c0t5000CCA07D293CA0d0'''

10. Check the progress with:

# '''zpool status'''
pool: rpool
state: ONLINE
scan: resilvered 75.8G in 8m24s with 0 errors on Wed Feb 27 11:39:38 2019

config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t5000CCA02F613ACCd0 ONLINE 0 0 0
c0t5000CCA07D293CA0d0 ONLINE 0 0 0

errors: No known data errors

11. Once the zpool is happy again, install the boot block on the new disk:

# '''installboot -f -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t5000CCA07D293CA0d0s0'''

Solaris 11: Useful commands

2018-09-27T03:26:22Z

Ppapa: /* Determining Oracle Solaris 11 OS Package Update Version */

== Determining Oracle Solaris 11 OS Package Update Version ==

To determine the package version of the Oracle Solaris 11 OS installed on your system:

pkg info kernel

== Check Available Versions ==

If you want to update your operating system release, check the available versions of the pkg:/entire incorporation package:

pkg list -af entire

VCenter Upgrade options for small sized vCenter deployments missing

2018-02-27T05:41:41Z

Ppapa:

So you've got the Migration Assistant to start and now you're half way through the migration assistant before you get this:

[[File:Vmwarex-largeonly.png]]

The ONLY option available to you is the "X-Large" deployment size. This is obviously less than ideal if you only want a small one! In my case, the issue was Postgres logs.

1. Log into the vCenter host via SSH and go to /storage/db/vpostgres/pg_log

2. Delete anything older than 30 days

find . -mtime +30 -exec rm {} \;

That was enough to convince the Upgrade utility that I could have a smaller one :-)

VCenter migration assistant failing to start (rejecting password)

2018-02-27T04:25:05Z

Ppapa:

I was trying to to a migration of a vCenter instance from v5.5 to v6.5. There was an Update Manager in play, which requires that you log into the Update Manager and run the Migration Assistant. However, when prompted for the "root" password, even though I had entered the correct password, repeeatedly, it failed, saying I had bad credentials. After much butting of my head against the wall, and Googling, I found out that to fix it, you need to run the following utility:

C:\Program Files (x86)\VMware\Infrastructure\Update Manager\VMwareUpdateManagerUtility.exe

then:

1. Login with SSO Admin (administrator@vsphere.local or similar)

2. Navigate to “re-register vCenter Server” and enter in the details again.

[[File:Vmwareupdatemanagerutility.png]]

3. Restart the VMware Update Server Service.

The Migration Assistant should now accept the correct password and start.