The mistake that is usually made with ICMP due to it being a jack of all trades protocols. It has numerous different functions. This has been specifically recognized in IPv6 where ICMP messages are clearly split into two categories: informational messages, and error messages.
Information ICMP messages include messages like ``echo request'' and ``echo reply'' and tend to seen by many security protocol designers as being the only kind that are relevant. If one is satisfied to deal only with informational datagrams, then it works just fine to treat them just like UDP or TCP datagrams.
Error messages are definitely different. To see this contrast them to with other traditional unicast protocols (ICMP informational messages, TCP and UDP datagrams).
Traditional protocols involve two hosts. They can be refered to as the "client" and "server", the "initiator" and "responder" or the "sender" and "receiver". For simplicity, call them "E1" and "E2".
Traditional protocols typically involve host E1 sending a datagram to host E2, and later on, E2 will send a datagram of the same protocol back to E1. While UDP and TCP use port numbers at each end to permit multiple flows between E1 and E2 (with responses being the same as the initial message, but with the source and destination port numbers exchanged), ICMP informational messages use message specific functions, and the answer to one type of datagram is often of another type (ICMP echo request/ICMP echo reply, ICMP router solicit/ICMP router advertisement, ICMP timestamp request/ICMP timestamp reply, ...)
ICMP error messages can involves just the two hosts. The ICMP unreachable message, code ICMP_UNREACH_PORT can be sent to E1 by E2. A key distinguishing factor is that this message is not sent by E2 in response to an ICMP datagram from E1. It is sent in response to a TCP or UDP datagram!
The break with traditional upper layer protocols is even more stark when one considers which hosts produce ICMP host unreachable, network unreachable, needs fragmentation, etc. In figure they are produced by R1, R2, R3 and in some configurations, by SG1 and SG2. This is most different as the selection of R1, R2 vs R4 is completely out of control of either E1 or E2. The ``correct'' set of routers which can legimately send ICMP host unreachable messages is nearly impossible for an end host to determine, and may change many times during the period of a connection.