tcpcb (was Re: sockbuf (was Re: BGL-free net stack))

[Aggelos Economopoulos]

On Friday 06 June 2008, Aggelos Economopoulos wrote:
[...]
> OK, same thing, but now it's the pcbs. TCP is "easy". The inpcb is inserted on
> a per-cpu hash table so that the corresponding protocol thread runs on the
> same cpu. Some tcpcb fields however, are accessed directly from user-thread
> context. The interesting fields are:
> 
> t_flags:
> 	need to do early copyin / delayed copyout in so_pr_ctloutput

So, I was thinking something like the following:

diff --git a/sys/kern/uipc_msg.c b/sys/kern/uipc_msg.c
index fde4c93..e786e02 100644
--- a/sys/kern/uipc_msg.c
+++ b/sys/kern/uipc_msg.c
@@ -35,6 +35,7 @@
 
 #include <sys/param.h>
 #include <sys/systm.h>
+#include <sys/kernel.h>
 #include <sys/msgport.h>
 #include <sys/protosw.h>
 #include <sys/socket.h>
@@ -379,22 +380,38 @@ so_pru_sopoll(struct socket *so, int events, struct ucred *cred)
 	return (error);
 }
 
+MALLOC_DEFINE(M_SOPT, "sopt", "sopt temp storage");
+
 int
 so_pr_ctloutput(struct socket *so, struct sockopt *sopt)
 {
-	return ((*so->so_proto->pr_ctloutput)(so, sopt));
 #ifdef gag	/* does copyin and copyout deep inside stack XXX JH */
+	return ((*so->so_proto->pr_ctloutput)(so, sopt));
+#else
 	struct netmsg_pr_ctloutput msg;
 	lwkt_port_t port;
 	int error;
+	void *uval;
+
+	uval = sopt->sopt_val;
 
-	port = so->so_proto->pr_mport(so, NULL);
+	/* we keep duplicate copies, but for option {s,g}etting who cares? */
+	sopt->sopt_val = kmalloc(sopt->sopt_valsize, M_SOPT, M_WAITOK);
+	error = copyin(uval, sopt->sopt_val, sopt->sopt_valsize);
+	if (error)
+		goto out;
+	port = so->so_proto->pr_mport(so, NULL, NULL, XXX);
 	netmsg_init(&msg.nm_netmsg, &curthread->td_msgport, 0,
 		    netmsg_pru_ctloutput);
 	msg.nm_prfn = so->so_proto->pr_ctloutput;
 	msg.nm_so = so;
 	msg.nm_sopt = sopt;
 	error = lwkt_domsg(port, &msg.nm_netmsg.nm_lmsg, 0);
+	if (error)
+		goto out;
+	error = copyout(sopt->sopt_val, uval, sopt->sopt_valsize);
+out:
+	kfree(sopt->sopt_val, M_SOPT);
 	return (error);
 #endif
 }

But before I update all callees to remove copy{in,out}(), does anybody have
any objections? Also, what should be the value of XXX? Perhaps ctloutput
belongs to pr_usrreqs...

Thanks,
Aggelos