RHEL 7 TimeZone lost after reboot

Hello community, I'm facing a strange behaviour on Red Hat 7.2

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)

I setup the TimeZone as:

# timedatectl set-timezone Europe/Rome
# timedatectl
      Local time: Fri 2021-06-18 14:55:27 CEST
  Universal time: Fri 2021-06-18 12:55:27 UTC
        RTC time: Fri 2021-06-18 12:55:27
       Time zone: Europe/Rome (CEST, +0200)
     NTP enabled: yes
NTP synchronized: no
RTC in local TZ: no
      DST active: yes

That's ok. But after reboot I have:

# timedatectl
      Local time: Fri 2021-06-18 12:59:44 UTC
  Universal time: Fri 2021-06-18 12:59:44 UTC
        RTC time: Fri 2021-06-18 12:59:44
       Time zone: UTC (UTC, +0000)
     NTP enabled: yes
NTP synchronized: no
RTC in local TZ: no
      DST active: n/a

So, TimeZone is lost after reboot. Could someone can drive me since I'm going crazy?
Thank you
Lucas

1 Like
/etc/localtime

is the file the OS looks at to set its timezone (not to be confused with /usr/share/zoneinfo/)

You probably need to establish a link to point to the desired timezone.

Ref:

Please give this a try and report back progress, or lack of.

I already did the commands reported on the link you posted. FOr RHEL7 they report:

timedatectl set-timezone Europe/Rome

After that I have:

/etc/localtime -> ../usr/share/zoneinfo/Europe/Rome

But after reboot I got:
etc/localtime -> /usr/share/zoneinfo/UTC

And did you do this?

# mv /etc/localtime /root/localtime.old
# ln -s /usr/share/zoneinfo/America/Los_Angeles /etc/localtime

So the first line moves the existing 'localtime' config elsewhere, and then the second line links 'localtime' to directly point to the required timezone, thereby, when the OS wants to read its timezone, it's going to be given the right one direct from /usr/share/zoneinfo options.

1 Like

Yes, I did it and reboot... now way :frowning:

You need to check that out again.

If the OS finds /etc/localtime is missing, it will default to UTC and that is what looks to be happening. Recheck that your symbolic link is in place properly.

What says:

# ls -l /etc/localtime
1 Like
# ls -l /etc/localtime
lrwxrwxrwx 1 root root 33 Jun 18 16:20 /etc/localtime -> ../usr/share/zoneinfo/Europe/Rome

This file is ok, but becomes (after reboot):

# ls -l /etc/localtime
lrwxrwxrwx 1 root root 23 Jun 18 14:10 /etc/localtime -> /usr/share/zoneinfo/UTC

I don't know if it's relevant, but after reboot in dmesg I can see:
Jun 18 14:11:01 dr01 systemd-timedated[2152]: RTC configured to UTC time.

Hmmmm.......is this a physical machine or a VM?

1 Like

It's a VM running under openstack

So it's a VM.

What timezone is the hypervisor configured to?

Perhaps the VM is ignoring NTP and getting its time set from the hypervisor.

2 Likes

Firstly, thank you for your time, I don't think it's a hypervisor issue (KVM however) since there are some other machine who are not affected to this proboem. I think the problem is on the OS, but I don't understand where :frowning:

Do you have another VM Redhat Enterprise 7.2 on the same hypervisor which doesn't have this issue?

The hypervisor can have an impact on the absolute clock time, but not on the timezone.
The timezone is a locally configured presentation.

Look at
https://man7.org/linux/man-pages/man1/timedatectl.1.html
and especially
https://man7.org/linux/man-pages/man1/systemd-firstboot.1.html

1 Like

Hello,

Could you try the following command, and see if this gets your system to respect and/or retain its timezone ?

timedatectl set-local-rtc yes

Hope this helps ! If not, let us know, and we can take things from there.

1 Like

Well, I setup set-local-rtc as suggested (I have temporarily disabled ntp service just to be sure it doesn't interfere with TZ):

timedatectl status
      Local time: Sat 2021-06-19 07:48:37 CEST
  Universal time: Sat 2021-06-19 05:48:37 UTC
        RTC time: Sat 2021-06-19 09:48:25
       Time zone: Europe/Rome (CEST, +0200)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: yes
      DST active: yes
 Last DST change: DST began at
                  Sun 2021-03-28 01:59:59 CET
                  Sun 2021-03-28 03:00:00 CEST
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2021-10-31 02:59:59 CEST
                  Sun 2021-10-31 02:00:00 CET

Warning: The system is configured to read the RTC time in the local time zone.
         This mode can not be fully supported. It will create various problems
         with time zone changes and daylight saving time adjustments. The RTC
         time is never updated, it relies on external facilities to maintain it.
         If at all possible, use RTC in UTC by calling
         'timedatectl set-local-rtc 0'.

But again, after reboot:

timedatectl status
      Local time: Sat 2021-06-19 10:17:20 UTC
  Universal time: Sat 2021-06-19 10:17:20 UTC
        RTC time: Sat 2021-06-19 10:17:20
       Time zone: UTC (UTC, +0000)
     NTP enabled: no
NTP synchronized: no
 RTC in local TZ: yes
      DST active: n/a

I'm surrendering :frowning:

I suspect out of bound modification of timezone settings.
Either by some script or other automatism tool.

Please check openstack cloud-init (you have cloud init log on the VM) for clues, and systemd-analyze
to check for custom script (/etc/rc files) or unit file which reconfigures timezone in boot process.

Hope that helps.
Regards
Peasant.

1 Like

Hi,

I think very thoroughly checking the settings for this VM on your virtualisation platform is definitely the way to go at this point, yes. It seems very likely that your hosting platform is re-setting the timezone on each re-start of the VM. I've seen that happen more often with containers than full-blown VMs, but it's not impossible, and is certainly worth looking into.

Alternatively, if the virtualisation platform is not to blame, then there is something else that runs on boot inside your VM that is changing the timezone back to UTC every time the server is re-started. As has been suggested by others, have a dig underneath /etc to see if you can see anything in there mentioning UTC. It's also worth checking out all the per-user crontabs, as well as the system-wide cron jobs undrneath the various /etc/cron* files and directories to see if there's a job there that would run on reboot that could be triggering this. All of this feels fairly unlikely to me, though - this definitely feels more like a VM management issue than anything else.

Anyway, good luck ! If I think of anything else I'll let you know, but hopefully you'll find a setting or checkbox or something on the VM platform that ultimately explains all this.

Thank you both, I also bet for something that changes the TZ at boot time, but honestly I don't know how to search inside the OS for this kind of references.
Could you please post some example on where/how to search for TZ change at boot? Thank you.

I tried this, but nothing found
find /etc -type f | xargs grep -l "UTC"

Hi,

Something like that approach would be fine, yes. But I don't think you're too likely to find anything, to be honest - I'd definitely focus on the VM management layer of this platform as being the source of the problem.

But if you do want to check through the whole of /etc for the string "UTC", you could simplify your approach there and do it all in one command ,like this:

find /etc -type f -exec grep -l UTC \{\} \;

By getting find to execute grep directly on every file it finds beneath /etc, you eliminate the need for the pipe into xargs. But the end result should be the same, in any case.

In terms of crontabs, check out the per-user crontab files in /var/spool/cron, as well as the system-wide global crontab file /etc/crontab. In addition to that, check the other scheduled crontab files in directories such as /etc/cron.d, /etc/cron.daily, /etc/cron.hourly, /etc/cron.monthly and /etc/cron.weekly. Although to be honest, anything in those should be caught by your find command anyway - if it directly mentions UTC of course, and doesn't simply run some other command somewhere else. So these are still worth a look even if your find turns up nothing.

There are a few other possibilities, but these should be enough to be getting on with. But as I say, I'd focus on the KVM layer first. This definitely smells more like a management layer somewhere re-setting your VM back to its "correct" settings when it re-starts, than anything else.

Yes, well, I still think that this is a platform/hypervisor problem.

After some searching I found this guy complaining of a similar problem in 2016:

https://www.hanbaoying.com/2016/06/22/vm-of-openstack-reset-to-UTC-time-when-reboot.html

but sods law that it's all in Chinese (at least I think it's Chinese) and I can't find a translate button (using Brave browser). Anyone on here understand Chinese? Of course, I don't even know whether there's a solution buried in here.

I could send the link to a good business friend in Taiwan but that will take time.

Thanks to anyone who can help.

1 Like