Help not trigger the trap

mainpid=$$
(trap "echo timeout" SIGTERM SIGKILL SIGINT; sleep 5; kill $mainpid) &
watchdogpid=$!

sleep 10 #do something 
kill $watchdogpid

I design a timeout function script
I don't see the echo "timeout" is on the screen

Why would you expect for this code to ever print timeout ?
Let's look at the likely sequence of operations:

parent: mainpid=PPID
parent: watchdogpid=CPID
parent: sleep 10
child: trap ...
child: sleep 5
child: kill $mainpid
child: exit
parent: killed by SIGTERM signal from child

so the child never received a signal that was caught by the trap command.

But, if you change the sleep 10 to sleep 1 then the sequence of operations is something like:

parent: manpid=PPID
parent: watchdogpid=CPID
parent: sleep 1
child: trap 'echo timeout' SIGTERM SIGKILL SIGINT
child: sleep 5
parent: kill $watchdogpid SIGTERM
child: sleep 5 terminated by SIGTERM signal from parent
child: echo timeout
child: kill $mainpid
child: exit
parent: exit (or terminate from SIGTERM from child

In this case, you see the output from the echo when the parent sends the child a SIGTERM signal.

Note that you can't successfully catch a SIGKILL. Did you perhaps mean to have this code instead:

mainpid=$$
trap "echo timeout; exit 1" SIGTERM SIGINT
(sleep 5; kill $mainpid) &
watchdogpid=$!

sleep 10 #do something 
kill $watchdogpid